Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-74868

Snapshot incorrectly labeled "QuiesceFailed" when freeze succeeds after retries (False Negative)

XMLWordPrintable

    • Quality / Stability / Reliability
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • None

      Description of problem:

      When taking a Virtual Machine Snapshot, if the initial file system freeze command fails (e.g., due to a timeout) but a subsequent retry succeeds, the final snapshot object is still tagged with QuiesceFailed.

      Version-Release number of selected component (if applicable):

      cnv-4.20 cnv4.21 

       

      Environment:

      • Storage Class: (e.g., ocs-storagecluster-ceph-rbd-virtualization)
      • Guest OS: Windows 11 (with Guest Agent installed)

      How reproducible:

      100% when the first freez attempt fail and the subsequent retry succeeds

      Steps to Reproduce:

      1.Deploy a Windows VM with the QEMU Guest Agent installed and running.
      2.Open an application (e.g., Notepad) and type text without saving to verify memory persistence later.
      3.Trigger a VirtualMachineSnapshot.
      4.Observe a scenario where the initial freeze takes longer than the default timeout (or fails temporarily), triggering the virt-controller retry mechanism.
      5.Wait for the snapshot phase to reach Succeeded.
      6. observe the QuiesceFailed tag in ui/ snapshot yaml
      

      Actual results:

      The virt-controller logs show that the initial freeze failed, followed by retries, and finally a successful freeze/unfreeze.
      
      The VirtualMachineSnapshot object has the QuiesceFailed indication.
      
      However, restoring the snapshot proves the data is Application Consistent (the Notepad text is preserved).
      
      oc get virtualmachinesnapshot -A -w
      NAMESPACE   NAME                                       SOURCEKIND       SOURCENAME        PHASE   READYTOUSE   CREATIONTIME   ERROR
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny                                       
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny                                       
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   InProgress   false                       
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   InProgress   false                       
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   InProgress   false                       Failed freezing vm windows-11-tiny: Internal error occurred: Put "https://10.131.0.49:8186/v1/namespaces/default/virtualmachineinstances/windows-11-tiny/freeze": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   InProgress   false                       Failed freezing vm windows-11-tiny: Internal error occurred: Put "https://10.131.0.49:8186/v1/namespaces/default/virtualmachineinstances/windows-11-tiny/freeze": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   InProgress   false                       Failed freezing vm windows-11-tiny: Internal error occurred: Put "https://10.131.0.49:8186/v1/namespaces/default/virtualmachineinstances/windows-11-tiny/freeze": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   InProgress   false                       Failed freezing vm windows-11-tiny: Internal error occurred: Put "https://10.131.0.49:8186/v1/namespaces/default/virtualmachineinstances/windows-11-tiny/freeze": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   InProgress   false                       
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   Succeeded    true         0s 
                  
      [cloud-user@ocp-psi-executor-xl ~]$ oc get virtualmachinesnapshot -A 
      NAMESPACE   NAME                                       SOURCEKIND       SOURCENAME        PHASE       READYTOUSE   CREATIONTIME   ERROR
      default     windows-11-tiny-snapshot-20251209-162038   VirtualMachine   windows-11-tiny   Succeeded   true         19s      
      

      Expected results:

      If the freeze operation eventually succeeds within the allowed retry window, the QuiesceFailed indication should not be present on the final snapshot object. The status should reflect the final successful state, not the transient errors.

      Additional info:
      controller log

      E1210 12:33:38.720577       1 util.go:66] "Unhandled Error" err="Operation cannot be fulfilled on virtualmachinesnapshots.snapshot.kubevirt.io \"windows-11-tiny-snapshot-20251210-143337\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251209-125737-1765278054374","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.757978Z","uid":"0abaa9ad-a8e3-4b3e-a3f8-cd90322a5468"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-131508-1765365724086","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.758114Z","uid":"0ad3f2da-07ac-4b8b-9dda-c682e9441a30"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133023-1765366445492","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.758242Z","uid":"40fc7d90-cca7-407d-84c8-177808d1de14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366863717","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.758362Z","uid":"16d5cd06-2db3-40f3-8ea8-b3cc9bdc1a14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-142025-1765369477934","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.759482Z","uid":"53494d8a-2bde-43c1-8717-b07611492a86"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366907276","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.758577Z","uid":"4addf6d0-226f-4a92-95ad-1d5def1aa626"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-135213-1765367697198","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.758630Z","uid":"b7bf1d25-d835-434c-8ae7-f0705dd45216"}
      {"component":"virt-controller","level":"info","msg":"Adding VM snapshot finalizer to windows-11-tiny","pos":"source.go:199","timestamp":"2025-12-10T12:33:38.757616Z"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251209-125737-1765278054374","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.858088Z","uid":"0abaa9ad-a8e3-4b3e-a3f8-cd90322a5468"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-131508-1765365724086","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.858248Z","uid":"0ad3f2da-07ac-4b8b-9dda-c682e9441a30"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133023-1765366445492","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.858310Z","uid":"40fc7d90-cca7-407d-84c8-177808d1de14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366863717","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.858361Z","uid":"16d5cd06-2db3-40f3-8ea8-b3cc9bdc1a14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366907276","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.858404Z","uid":"4addf6d0-226f-4a92-95ad-1d5def1aa626"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-135213-1765367697198","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.858482Z","uid":"b7bf1d25-d835-434c-8ae7-f0705dd45216"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-142025-1765369477934","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:38.858564Z","uid":"53494d8a-2bde-43c1-8717-b07611492a86"}
      {"component":"virt-controller","level":"info","msg":"Freeze VMI windows-11-tiny","pos":"virtualmachineinstance_expansion.go:146","timestamp":"2025-12-10T12:33:38.880658Z"}
      {"component":"virt-controller","level":"info","msg":"Freezing vmi windows-11-tiny took 10.029890952s","pos":"util.go:44","timestamp":"2025-12-10T12:33:48.910549Z"}
      {"component":"virt-controller","level":"error","msg":"Failed freezing vm windows-11-tiny: Internal error occurred: Put \"https://10.129.3.139:8186/v1/namespaces/default/virtualmachineinstances/windows-11-tiny/freeze\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","pos":"source.go:458","timestamp":"2025-12-10T12:33:48.910734Z"}
      {"component":"virt-controller","level":"info","msg":"Freeze VMI windows-11-tiny","pos":"virtualmachineinstance_expansion.go:146","timestamp":"2025-12-10T12:33:48.925338Z"}
      {"component":"virt-controller","level":"info","msg":"Freezing vmi windows-11-tiny took 5.715101484s","pos":"util.go:44","timestamp":"2025-12-10T12:33:54.640439Z"}
      {"component":"virt-controller","level":"info","msg":"Attempting to create VolumeSnapshot vmsnapshot-32d85f97-dfa6-42e4-ba4e-01b8667c8f1a-volume-rootdisk","pos":"snapshot.go:517","timestamp":"2025-12-10T12:33:54.640496Z"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-142025-1765369477934","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:54.687787Z","uid":"53494d8a-2bde-43c1-8717-b07611492a86"}
      {"component":"virt-controller","level":"info","msg":"Unfreeze VMI windows-11-tiny","pos":"virtualmachineinstance_expansion.go:170","timestamp":"2025-12-10T12:33:59.181181Z"}
      {"component":"virt-controller","level":"info","msg":"Unfreezing vmi windows-11-tiny took 80.646807ms","pos":"util.go:44","timestamp":"2025-12-10T12:33:59.261828Z"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251209-125737-1765278054374","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.339754Z","uid":"0abaa9ad-a8e3-4b3e-a3f8-cd90322a5468"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-131508-1765365724086","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.339879Z","uid":"0ad3f2da-07ac-4b8b-9dda-c682e9441a30"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133023-1765366445492","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.340401Z","uid":"40fc7d90-cca7-407d-84c8-177808d1de14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366863717","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.340476Z","uid":"16d5cd06-2db3-40f3-8ea8-b3cc9bdc1a14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366907276","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.340541Z","uid":"4addf6d0-226f-4a92-95ad-1d5def1aa626"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-135213-1765367697198","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.340628Z","uid":"b7bf1d25-d835-434c-8ae7-f0705dd45216"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-142025-1765369477934","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.340673Z","uid":"53494d8a-2bde-43c1-8717-b07611492a86"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-131508-1765365724086","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.358259Z","uid":"0ad3f2da-07ac-4b8b-9dda-c682e9441a30"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366863717","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.358346Z","uid":"16d5cd06-2db3-40f3-8ea8-b3cc9bdc1a14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133858-1765366907276","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.358402Z","uid":"4addf6d0-226f-4a92-95ad-1d5def1aa626"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-135213-1765367697198","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.358497Z","uid":"b7bf1d25-d835-434c-8ae7-f0705dd45216"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-133023-1765366445492","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.358438Z","uid":"40fc7d90-cca7-407d-84c8-177808d1de14"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-142025-1765369477934","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.358552Z","uid":"53494d8a-2bde-43c1-8717-b07611492a86"}
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251209-125737-1765278054374","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:33:59.358222Z","uid":"0abaa9ad-a8e3-4b3e-a3f8-cd90322a5468"}
      E1210 12:33:59.366946       1 util.go:66] "Unhandled Error" err="Operation cannot be fulfilled on virtualmachines.kubevirt.io \"windows-11-tiny\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
      {"component":"virt-controller","kind":"","level":"info","msg":"Updating VirtualMachineRestore","name":"restore-windows-11-tiny-snapshot-20251210-142025-1765369477934","namespace":"default","pos":"restore.go:190","timestamp":"2025-12-10T12:35:03.966092Z","uid":"53494d8a-2bde-43c1-8717-b07611492a86"}
      

              ngavrilo@redhat.com Natalie Gavrielov
              rh-ee-ahafe Ahmad Hafi
              Kedar Bidarkar Kedar Bidarkar
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: