Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-38746

Cannot live migrate VMs with bridge interface after performing a 4.12 to 4.14 EUS upgrade

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • CNV Documentation
    • None
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • ---
    • ---
    • High
    • No

      Description of problem:

      During the EUS upgrade, the worker mcp will be paused. So the worker nodes will stay in 4.12 until the master and other operators are upgraded to 4.14. The OpenShift Virtualization will be also get upgraded to 4.14 before worker nodes which also upgrades kube-cni-linux-bridge-plugin. This installs new versions of the cnv-bridge and cnv-tuning in the worker nodes:

      # oc logs kube-cni-linux-bridge-plugin-2rhm6
Installing bridge and tuning CNIs
Checking bridge and tuning CNIs deployment on node
/opt/cni/bin/cnv-bridge: OK
/opt/cni/bin/cnv-tuning: OK
Entering sleep... (success)  

      Since the worker nodes are still 8.x version, the bridge binary will fail in these node because it don't have the new GLIBC:

      openshift-worker-mytemp-0 ~]# /var/lib/cni/bin/cnv-bridge
/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/lib/cni/bin/cnv-bridge)
/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/lib/cni/bin/cnv-bridge)

openshift-worker-mytemp-0 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux CoreOS release 4.12

      Now when the user upgrade the worker pool, the VM live migration will fail and the destination virt-launcher pod will be stuck in ContainerCreating with error below:

      4m10s       Warning   FailedCreatePodSandBox       pod/virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm_nijin-cnv_7ac1c96b-7b9f-4da4-8712-8d1b914d7371_0(88ece225a8dec08a2ec02207ddb55a4ff843af486aad6ad1c42e574ddf2ae073): error adding pod nijin-cnv_virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [nijin-cnv/virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm/7ac1c96b-7b9f-4da4-8712-8d1b914d7371:br1]: error adding container to network "br1": netplugin failed: "/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/lib/cni/bin/cnv-bridge)\n/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/lib/cni/bin/cnv-bridge)\n"

      Version-Release number of selected component (if applicable):

      4.12 to 4.14 EUS Upgrade

      How reproducible:

      100%

      Steps to Reproduce:

      1. Install 4.12 and subscribe to eus channel.
      2. Start VMs with bridge interfaces.
      3. Follow EUS-to-EUS update to upgrade the cluser.
      4. When worker pool is resumed and upgraded, the VMs will fail to live migrate. 

      Actual results:

      Cannot live migrate VMs with bridge interface while performing a 4.12 to 4.14 EUS upgrade.

      Expected results:

      Provide option for 4.12 to 4.14 EUS update without VM downtime.

      Additional info:

       

              ctomasko Catherine Tomasko
              rhn-support-nashok Nijin Ashok
              Yossi Segev Yossi Segev
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: