Upgrade from 4.12.x to 4.13.x is impacted due systemd-network-generator.service service failure

    4.13.35

    Easily

    1. Upgrade dual stack ovn cluster from 4.12.x to 4.13.x
    2. Make sure the kernel cmdline argument has ip=dhcp,dhcp6    
    
    

OVNkube-node pods are failing as it cannot get IPv6 address:


 2024-03-19T08:43:26.482301725Z I0319 08:43:26.482294 1335201 ovs.go:204] Exec(10): stderr: ""

2024-03-19T08:43:26.529778064Z I0319 08:43:26.529755 1335201 gateway_init.go:261] Initializing Gateway Functionality

2024-03-19T08:43:26.530176353Z I0319 08:43:26.530130 1335201 
gateway_localnet.go:174] Node local addresses initialized to: 
map[10.130.2.2:{10.130.2.0 fffffe00} 10.63.100.145:{10.63.100.128 
ffffffe0} 10.63.107.135:{10.63.107.128 ffffffe0} 127.0.0.1:{127.0.0.0 
ff000000} 172.22.0.212:{172.22.0.0 ffffff00} 
214.12.176.168:{214.12.176.160 ffffffe0} ::1:{::1 
ffffffffffffffffffffffffffffffff} fd01:0:0:2::2:{fd01:0:0:2:: ffffffffffffffff0000000000000000} fe80::1270:fdff:fed8:6d20:{fe80:: ffffffffffffffff0000000000000000} fe80::38cf:dff:feb0:87fb:{fe80:: ffffffffffffffff0000000000000000} fe80::928d:6eff:feff:4a8a:{fe80:: ffffffffffffffff0000000000000000} fe80::d865:62ff:fe99:4695:{fe80:: ffffffffffffffff0000000000000000}]

2024-03-19T08:43:26.530235846Z I0319 08:43:26.530227 1335201 
helper_linux.go:78] Provided gateway interface "br-ex", found as index: 
15

2024-03-19T08:43:26.530314081Z I0319 08:43:26.530306 1335201 
helper_linux.go:103] Found default gateway interface br-ex 10.63.100.129

2024-03-19T08:43:26.530352839Z I0319 08:43:26.530345 1335201 
helper_linux.go:78] Provided gateway interface "br-ex", found as index: 
15

2024-03-19T08:43:26.530532039Z I0319 08:43:26.530524 1335201 metrics.go:504] Stopping metrics server 127.0.0.1:29103

2024-03-19T08:43:26.530549065Z I0319 08:43:26.530537 1335201 reflector.go:227] Stopping reflector *v1.Node (0s) from k8s.io/client-go/informers/factory.go:150

2024-03-19T08:43:26.530580063Z F0319 08:43:26.530570 1335201 
ovnkube.go:137] failed to start node network manager: failed to start 
default node network controller: failed to find IPv6 address on 
interface br-ex

The ovn issue is not related to systemd-network-generator service. Removing ip=dhcp,dhcp6 from kernel cmdline helps the service to start successfully. 


Below services are failed when the nodes are updated to RHEL9:

● crio-wipe.service                  loaded failed failed CRI-O Auto Update Script

● NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online

● systemd-network-generator.service  loaded failed failed Generate network units from Kernel command line


systemd-network-generator[1530953]: Failed to parse kernel command line: Invalid argume>
Mar 19 13:08:37  systemd[1]: systemd-network-generator.service: Main process exited, code=exited, status>
Subject: Unit process exited

    The upgrade is smooth and there are no service failures.

We are hitting a known issue mentioned in below JIRA:

systemd-network-generator.service fails - "Failed to start Generate network units from Kernel command line"
https://issues.redhat.com/browse/RHELPLAN-156191

systemd-network-generator.service: Failed to parse kernel command line
https://issues.redhat.com/browse/OCPBUGS-12867 

The affected system has ip=dhcp,dhcp6

$ cat proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-57123c5c4e8a5d5cf5c426d17484390a6d8f707e07b0a3da97d8791485774728/vmlinuz-5.14.0-284.52.1.el9_2.x86_64 ignition.platform.id=metal ostree=/ostree/boot.1/rhcos/57123c5c4e8a5d5cf5c426d17484390a6d8f707e07b0a3da97d8791485774728/0 ip=dhcp,dhcp6 root=UUID=49f6aa83-dc80-4ccd-a662-ca706cd4889d rw rootflags=prjquota boot=UUID=ce1a278f-7fe7-41f5-aab7-6e6a003e8474 systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=1