In OVN-Kubernetes environment, OpenShift RHEL8 worker nodes cannot upgrade due to the networking broken after yum allowerasing upgrade from openvswitch2.15 to 3.1 version

OpenShift 4.12 version or later

In OCP4.12.13 OVN-Kubernetes plugin cluster is installed earyly and the rpm package of openvswitch is staying at 2.15 version.

The RHEL8 worker would want to use the ansible-playbook to upgrade the latest 4.12.z or 4.13.41.

# cd /usr/share/ansible/openshift-ansible/
# ansible-playbook -vvv -i /root/rhel-worker/inventory playbooks/upgrade.yml

ansible-playbook tasks will be failed in TASK [openshift_node : Pull MCD image] due to the networking broken after yum allowerasing upgrade from openvswitch2.15 to openvswitch3.1 version step of 'yum install openvswitch3.1 --allowerasing -y' after TASK [openshift_node : Install openshift packages]

Details:
https://github.com/openshift/openshift-ansible/blob/release-4.13/roles/openshift_node/tasks/upgrade.yml#L28-L29
# Upgrade Node Packages
- import_tasks: install.yml

https://github.com/openshift/openshift-ansible/blob/release-4.13/roles/openshift_node/tasks/install.yml#L90-L100
- block:
  - name: Install openshift packages
    dnf:
      name: "{{ openshift_packages }}"
      state: latest
      allowerasing: true <- 'allowerasing' parameter causes the networking broken due to openvswitch version is changed
      disable_gpg_check: true
    async: 3600
    poll: 30
    register: result
    until: result is succeeded

https://github.com/openshift/openshift-ansible/blob/release-4.13/roles/openshift_node/defaults/main.yml#L13-L87
openshift_packages: "{{ (openshift_node_packages + openshift_node_support_packages) | join(',') }}"

openshift_node_packages:
  - conmon
  - cri-o-{{ crio_latest }}
  - cri-tools
  - openshift-clients-{{ l_cluster_version }}*
  - openshift-hyperkube-{{ l_cluster_version }}*
  - podman
  - runc

openshift_node_support_packages: "{{
  openshift_node_support_packages_base +
  openshift_node_support_packages_by_os_major_version[ansible_distribution_major_version] +
  openshift_node_support_packages_by_arch[ansible_architecture] }}"

openshift_node_support_packages_base:
  - kernel
  - systemd
  - selinux-policy-targeted
  - setools-console
  - dracut-network
  - passwd
  - openssh-server
  - openssh-clients
  - skopeo
  - containernetworking-plugins
  - nfs-utils
  - NetworkManager
  - NetworkManager-ovs  # https://bugzilla.redhat.com/show_bug.cgi?id=1884095
  - dnsmasq
  - lvm2
  - iscsi-initiator-utils
  - sg3_utils
  - device-mapper-multipath
  - xfsprogs
  - e2fsprogs
  - mdadm
  - cryptsetup
  - chrony
  - logrotate
  - sssd
  - shadow-utils
  - sudo
  - coreutils
  - less
  - tar
  - xz
  - gzip
  - bzip2
  - rsync
  - tmux
  - nmap-ncat
  - net-tools
  - bind-utils
  - strace
  - bash-completion
  - vim-minimal
  - nano
  - authconfig
  - iptables-services
  - cifs-utils  # https://bugzilla.redhat.com/show_bug.cgi?id=1827982
  - jq
  - libseccomp

openshift_node_support_packages_by_os_major_version:
  "7":
    - openvswitch2.13
    - policycoreutils-python
    - bridge-utils
    - container-storage-setup
    - ceph-common
  "8":
    - openvswitch3.1 <- Here upgrade to openvswitch3.1 from 2.15 will be networking broken
    - policycoreutils-python-utils
Steps to Reproduce:

1. In OVN-Kubernetes Environment, keep the RHEL8 worker installed the old version openvswitch 2.15 and reboot the machine ensure the networking everything is working.

2. Use the ansible-playbook for upgrade the RHEL8 worker nodes to 4.13.41 version.
# cd /usr/share/ansible/openshift-ansible/
# ansible-playbook -vvv -i /root/rhel-worker/inventory playbooks/upgrade.yml

3. ansible-playbook tasks will be failed in TASK [openshift_node : Pull MCD image] due to the networking broken after yum allowerasing upgrade from openvswitch2.15 to openvswitch3.1 version step of 'yum install openvswitch3.1 --allowerasing -y' after TASK [openshift_node : Install openshift packages]

4. The upgrading targets RHEL8 worker nodes become NotReady state due to the networking broken and ansible-playbook cannot finish all the upgrading tasks.

ansible-playbook tasks pending on TASK [openshift_node : Pull MCD image] due to the OVN-Kubernetes networking broken after upgraded openvswitch2.15 to 3.1 version

ansible-playbook tasks could complete all the tasks and fix the OVN-Kubernetes networking broken after upgrading openvswitch2.15 to 3.1 version

The customer cannot upgrade OVN-Kubernetes plugin RHEL8 worker nodes due to the networking broken, please provide the workaround to assist the customer RHEL8 worker nodes upgrading or fix the broken networking.