-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
3
-
False
-
-
False
-
None
-
---
-
---
-
-
Storage Core Sprint 262, Storage Core Sprint 263
-
High
-
None
Description of problem:
During a snapshot, if an fsfreeze fails, for example due to https://access.redhat.com/solutions/7087713, the system will just continue to loop trying to fsfreeze again and again, until it reaches the snapshot deadline. {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.088429Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.088465Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.126064Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.126094Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.165165Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.165206Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.217425Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.217457Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.288655Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.288684Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.402307Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.402336Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.593687Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.593725Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.949590Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.949619Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:15.625161Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:15.625201Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:17.074487Z"} {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:17.074518Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} {"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:208","timestamp":"2024-09-18T05:50:19.537987Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"} ..... repeats over and over
But the snapshot fails with "deadline exceeded", which is not really showing the problem to the user.
# oc get virtualmachinesnapshot snapshot-aquamarine-hare-33 -o yaml apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineSnapshot metadata: creationTimestamp: "2024-09-18T05:50:13Z" finalizers: - snapshot.kubevirt.io/vmsnapshot-protection generation: 4 name: snapshot-aquamarine-hare-33 namespace: homelab ownerReferences: - apiVersion: kubevirt.io/v1 blockOwnerDeletion: false kind: VirtualMachine name: rhel-8 uid: 1bd72695-04e8-450e-9998-46b62dd92895 resourceVersion: "1011220" uid: c349c24a-7d0a-4db2-b57d-84c947388328 spec: source: apiGroup: kubevirt.io kind: VirtualMachine name: rhel-8 status: conditions: - lastProbeTime: null lastTransitionTime: "2024-09-18T05:55:13Z" reason: snapshot deadline exceeded status: "False" type: Progressing - lastProbeTime: null lastTransitionTime: "2024-09-18T05:50:14Z" reason: Not ready status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2024-09-18T05:55:13Z" reason: snapshot deadline exceeded status: "True" type: Failure indications: - Online - GuestAgent phase: Failed readyToUse: false sourceUID: 1bd72695-04e8-450e-9998-46b62dd92895
Version-Release number of selected component (if applicable):
4.16.2
How reproducible:
100%
Steps to Reproduce:
Inside the Guest: $ mount -o bind /var/ /mnt/test/ $ chcon -t rpm_var_lib_t /mnt/test $ mount -o remount,ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota,suid /mnt/test/ eclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
Actual results:
"Deadline exceeded", doesn't say where we were stuck at when the deadline was reached.
Expected results:
Proper error propagation, tell the user there was a problem freezing the guest fs
- is related to
-
CNV-32664 [2237678] VM snapshot is failing during fsfreeze with permission denied error
- Closed
- mentioned on