-
Bug
-
Resolution: Unresolved
-
Normal
-
CNV v4.16.0, CNV v4.16.5
-
None
-
Incidents & Support
-
1
-
False
-
-
False
-
None
-
-
CNV Storage 269, CNV Storage 270
-
Important
-
None
Description of problem:
We have seen that OADP backups of Windows 2022 and 2016 VMs fail from time to time with an error when freezing the fs:
time="2024-12-23T12:51:08Z" level=info msg="stderr: {\"component\":\"freezer\",\"level\":\"info\",\"msg\":\"Starting...\",\"pos\":\"main.go:59\",\"timestamp\":\"2024-12-23T12:51:03.121845Z\"}\n{\"component\":\"freezer\",\"level\":\"info\",\"msg\":\"Guest agent version is 108.0.1\",\"pos\":\"main.go:102\",\"timestamp\":\"2024-12-23T12:51:03.125041Z\"}\n{\"component\":\"freezer\",\"level\":\"error\",\"msg\":\"Freezeing VMI failed\",\"pos\":\"main.go:112\",\"reason\":\"server error. command Freeze failed: \\\"LibvirtError(Code=86, Domain=10, Message='Guest agent is not responding: Guest agent not available for now')\\\"\",\"timestamp\":\"2024-12-23T12:51:08.145052Z\"}\n" backup=openshift-adp/backup-case-03980918-04 hookCommand="[/usr/bin/virt-freezer --freeze --name windows-2k22-virtio-crimson-worm-76 --namespace default]" hookContainer=compute hookName="<from-annotation>" hookOnError=Fail hookPhase=pre hookSource=annotation hookTimeout="{30s}" hookType=exec logSource="/remote-source/velero/app/pkg/podexec/pod_command_executor.go:181" name=virt-launcher-windows-2k22-virtio-crimson-worm-76-bmxnl namespace=default resource=pods
time="2024-12-23T12:51:08Z" level=error msg="Error executing hook" backup=openshift-adp/backup-case-03980918-04 error="command terminated with exit code 1" hookPhase=pre hookSource=annotation hookType=exec logSource="/remote-source/velero/app/internal/hook/item_hook_handler.go:239" name=virt-launcher-windows-2k22-virtio-crimson-worm-76-bmxnl namespace=default resource=pods
The problem is that the guest agent call 'guest-fsfreeze-freeze' returns error:
$ oc exec virt-launcher-windows-2k22-virtio-crimson-worm-76-z4mhk -- virsh qemu-agent-command 1 '{"execute": "guest-fsfreeze-freeze"}'
error: Guest agent is not responding: Guest agent not available for now
command terminated with exit code 1
Version-Release number of selected component (if applicable):
kubevirt-hyperconverged-operator.v4.16.5 QEMU Guest Agent 108.0.1
How reproducible:
Most of the time, but not always.
Steps to Reproduce:
1. Install Windows Server 2022 with virtio devices (windows.2k22.virtio Preference)
2. Install virtio drivers and qemu-ga from the ISO provided by OCP
3. Run fsfreeze in the guest. Can also be triggered by creating a VM snapshot:
$ oc exec virt-launcher-windows-2k22-virtio-crimson-worm-76-z4mhk -- virsh qemu-agent-command 1 '{"execute": "guest-fsfreeze-freeze"}'
error: Guest agent is not responding: Guest agent not available for now
command terminated with exit code 1
$ oc exec virt-launcher-windows-2k22-virtio-crimson-worm-76-z4mhk -- virsh qemu-agent-command 1 '{"execute": "guest-fsfreeze-status"}'
{"return":"frozen"}
$ oc exec virt-launcher-windows-2k22-virtio-crimson-worm-76-z4mhk -- virsh qemu-agent-command 1 '{"execute": "guest-fsfreeze-thaw"}'
{"return":3}
Actual results:
'guest-fsfreeze-freeze' call returns error 1 Apparently the filesystems are frozen correctly and no errors can be seen in the Windows event log.
Expected results:
No errors in the freeze call.
Additional info: