Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-55362

[CNV-4.19][Tracker Bug] Live migration after workload update fails with operation failed: guest CPU doesn't match specification: missing features: waitpkg

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • CNV v4.19.0
    • CNV v4.19.0
    • CNV Virt-Cluster
    • None
    • Incidents & Support
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • CNV Virt-Cluster Sprint 267
    • Moderate
    • Customer Reported
    • None

      Description of problem:

      After upgrade of CNV from v4.15.8 to v4.16.5, live migration of VM for workload update fails with 'operation failed: guest CPU doesn't match specification: missing features: waitpkg'
      ~~~
      2025-01-22T17:39:25.458557070Z {"component":"virt-launcher","level":"info","msg":"2025-01-22T17:39:24.907135Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.07H:ECX.waitpkg [bit 5]","subcomponent":"qemu","timestamp":"2025-01-22T17:39:25.458323Z"}
      2025-01-22T17:39:25.941847222Z {"component":"virt-launcher","level":"error","msg":"operation failed: guest CPU doesn't match specification: missing features: waitpkg","pos":"virCPUx86UpdateLive:3107","subcomponent":"libvirt","thread":"33","timestamp":"2025-01-22T17:39:25.941000Z"}
      ~~~
      The only way to resolve this is to stop/start the VM.  After stopping and starting the VM, a migration succeeds, so the issue is only seen during the upgrade window. 
      The failing migration source pod:
      
      2025-01-11T20:32:45.827149786Z {"component":"virt-launcher","level":"info","msg":"libvirt version: 9.0.0, package: 10.10.el9_2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2024-09-12-10:51:56, )","subcomponent":"libvirt","thread":"45","timestamp":"2025-01-11T20:32:45.826000Z"}2025-01-22T17:40:17.990830574Z {"component":"virt-launcher","level":"error","msg":"operation failed: guest CPU doesn't match specification: missing features: waitpkg","pos":"virNetClientProgramDispatchError:172","subcomponent":"libvirt","thread":"31","timestamp":"2025-01-22T17:40:17.990000Z"}
      
      2025-01-22T17:40:17.997873194Z {"component":"virt-launcher","kind":"","level":"error","msg":"Live migration failed.","name":"ood-1","namespace":"example","pos":"live-migration-source.go:969","reason":"error encountered during MigrateToURI3 libvirt api call: virError(Code=9, Domain=31, Message='operation failed: guest CPU doesn't match specification: missing features: waitpkg')","timestamp":"2025-01-22T17:40:17.997579Z","uid":"369eb6f6-f134-4a6d-9f8c-a630a2eb06cc"}
      
      2025-01-22T17:40:18.083291028Z {"component":"virt-launcher","kind":"","level":"error","msg":"Received a live migration error. Will check the latest migration status.","name":"ood-1","namespace":"example","pos":"live-migration-source.go:714","reason":"error encountered during MigrateToURI3 libvirt api call: virError(Code=9, Domain=31, Message='operation failed: guest CPU doesn't match specification: missing features: waitpkg')","timestamp":"2025-01-22T17:40:18.082943Z","uid":"369eb6f6-f134-4a6d-9f8c-a630a2eb06cc"}
      
      The xml generated has:<feature policy=\"require\" name=\"waitpkg\">
      
      
      cat /proc/cpuinfo on the source and dest nodes both show  (abbreviated) 
      flags		: waitpkg

      Version-Release number of selected component (if applicable):

      source pod for migration:  libvirt version: 9.0.0, package: 10.10.el9_2 qemu version: 7.2.0qemu-kvm-7.2.0-14.el9_2.15,  kernel: 5.14.0-427.49.1.el9_4.x86_64
      
      dest pod for migration:  libvirt version: 10.0.0, package: 6.12.el9_4  qemu version: 8.2.0qemu-kvm-8.2.0-11.el9_4.8 kernel: 5.14.0-427.49.1.el9_4.x86_64,

      How reproducible:

      Customer upgraded from CNV v4.15.8 to v4.16.5

      Expected results:

      Workload Update Live migrations succeed.
       

        1. debugLogsForLibvirt.txt
          62 kB
          Barak Mordehai
        2. domain.xml
          9 kB
          Jiri Denemark
        3. qemu-destination.txt
          6 kB
          Jiri Denemark
        4. qemu-source.txt
          6 kB
          Jiri Denemark

              bmordeha@redhat.com Barak Mordehai
              shaselde@redhat.com Sean Haselden
              Igor Mammedov
              Akriti gupta Akriti gupta
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: