Upgrade is stuck due to network cluster operator. Network cluster operator is failing as couple of ovnkube-node PODs are failing with below error in ovnkube-node container.

~~~
2023-04-21T15:07:31.178062995Z F0421 15:07:31.178047  181667 ovnkube.go:133] error waiting for node readiness: failed to set the node masquerade route to OVN: could not find node IPv6 address to configure OVN masquerade route, addresses: [{Type:InternalIP Address:<IPv4 IP>} {Type:Hostname Address:<hostname>}]

~~~

This issue may come up if IPv6 IP assignment to the primary interface is delayed for some reason.  As a result IPv6 IP does not get written to 20-ndenet.conf created by nodeip-cofiguratio service and in turn we do not get IPv6 IP in status of node custom resource. I have followed below workarounds mentioned in bug[1].

- Changed NM_ONLINE_TIMEOUT of NetworkManager-wait-online service to 300 second so that IPv6 Ip gets assigned to primary interface before starting nodeip-configuration service. NetworkManager-wait-online service is still failing after 5 minutes.
- Created an environment file for kubelet to refer and hard coded both IPv4 and Ipv6 IP. After rebooting the node, I do not see IPv6 IP in status section of node custom resource.

Next action plan:

- Add 'may-fail=false' under IPv6 in nmconnection file of bond0. Could not apply this yet as machine config operator is in Degraded state for a different reason and the nmconnection file is being managed by machine config operator.

I am not sure why nodeip-configuration service is not writing IPv6 IP to environment file 20-nodenet.conf even after 300 second delay was induced due to 'NM_ONLINE_TIMEOUT' in NetworkManager-wait-online service. I am also not sure why IPv6 IP is not showing in the status section even after mentioning both IPv4 and IPv6 Ip at 98-nodenet-override.conf in /etc/systemd/system/kubelet.service.d/ directory.

[1] - https://issues.redhat.com/browse/OCPBUGS-6009 

Openshift 4.12.4

Not Sure

1.
2.
3.

Upgrade is stuck due to this reason.

Upgrade should proceed

Will add in comments section