Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-62148

Migration target pod shutdown during migration

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • CNV v4.20.0
    • CNV v4.19.0
    • CNV Virt-Node
    • None
    • Quality / Stability / Reliability
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • Critical
    • None

      Description of problem:

      VM migration target pod shutdown during migration
      
      Migrated ~2000 VMs, each time approximately 10-12 failed to migrate

      Version-Release number of selected component (if applicable):

      v4.19.0.rhel9-138 IIB:965139

      How reproducible:

      100%

      Steps to Reproduce:

      1. Create VMs (tested with ~2000 vms)
      2. Migrate VMs until error occurs
      

      Actual results:

      virt-controller: target pod default/virt-launcher-vm-instancetype-cirros-test-0773-hl7ch shutdown during migration

      Expected results:

      migrations successful

      Additional info:

      virt-launcher:
      
      {"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:88","timestamp":"2025-05-19T15:50:30.074539Z"}
      {"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:91","timestamp":"2025-05-19T15:50:30.074579Z"}
      {"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu+unix:///session?socket=/var/run/libvirt/virtqemud-sock","pos":"libvirt.go:547","timestamp":"2025-05-19T15:50:30.074884Z"}
      {"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory')","pos":"libvirt.go:555","timestamp":"2025-05-19T15:50:30.075154Z"}
      {"component":"virt-launcher","level":"info","msg":"libvirt version: 10.10.0, package: 7.1.el9_6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2025-03-07-07:48:15, )","subcomponent":"libvirt","thread":"45","timestamp":"2025-05-19T15:50:30.090000Z"}
      {"component":"virt-launcher","level":"info","msg":"hostname: vm-instancetype-cirros-test-0773","subcomponent":"libvirt","thread":"45","timestamp":"2025-05-19T15:50:30.090000Z"}
      {"component":"virt-launcher","level":"error","msg":"internal error: Unable to get session bus connection: Cannot spawn a message bus without a machine-id: Invalid machine ID in /var/lib/dbus/machine-id or /etc/machine-id","pos":"virGDBusGetSessionBus:126","subcomponent":"libvirt","thread":"45","timestamp":"2025-05-19T15:50:30.090000Z"}
      {"component":"virt-launcher","level":"error","msg":"internal error: Unable to get system bus connection: Could not connect: No such file or directory","pos":"virGDBusGetSystemBus:99","subcomponent":"libvirt","thread":"45","timestamp":"2025-05-19T15:50:30.090000Z"}
      {"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:563","timestamp":"2025-05-19T15:50:30.576898Z"}
      {"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:573","timestamp":"2025-05-19T15:50:30.579386Z"}
      {"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:76","timestamp":"2025-05-19T15:50:30.579503Z"}
      panic: timed out waiting for domain to be defined
      {"component":"virt-launcher-monitor","level":"info","msg":"Reaped Launcher main pid","pos":"virt-launcher-monitor.go:199","timestamp":"2025-05-19T15:56:03.585368Z"}
      {"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 13 with status 512","pos":"virt-launcher-monitor.go:202","timestamp":"2025-05-19T15:56:03.585498Z"}
      {"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:216","timestamp":"2025-05-19T15:56:03.585517Z"}
      {"component":"virt-launcher-monitor","level":"info","msg":"virt-launcher-monitor: Exiting...","pos":"virt-launcher-monitor.go:145","timestamp":"2025-05-19T15:56:11.397269Z"}
      

       

              ibezukh Igor Bezukh
              rhn-support-sbennert Sarah Bennert
              Denys Shchedrivyi Denys Shchedrivyi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: