Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-48733

Error not properly propagated on fsfreeze failure

XMLWordPrintable

    • Storage Core Sprint 262, Storage Core Sprint 263
    • High
    • None

      Description of problem:

      
      During a snapshot, if an fsfreeze fails, for example due to https://access.redhat.com/solutions/7087713, the system will just continue to loop trying to fsfreeze again and again, until it reaches the snapshot deadline.
      
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.088429Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.088465Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.126064Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.126094Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.165165Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.165206Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.217425Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.217457Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.288655Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.288684Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.402307Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.402336Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.593687Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.593725Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:14.949590Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:14.949619Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:15.625161Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:15.625201Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","level":"error","msg":"Failed to freeze vmi, virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","pos":"manager.go:1629","timestamp":"2024-09-18T05:50:17.074487Z"}
      {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:269","reason":"virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /mnt/test: Permission denied')","timestamp":"2024-09-18T05:50:17.074518Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      {"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"rhel-8","namespace":"homelab","pos":"server.go:208","timestamp":"2024-09-18T05:50:19.537987Z","uid":"ce92b1aa-79e5-4136-a220-1ed24e10a819"}
      .....
      repeats over and over
      

      But the snapshot fails with "deadline exceeded", which is not really showing the problem to the user.

      # oc get virtualmachinesnapshot snapshot-aquamarine-hare-33 -o yaml
      apiVersion: snapshot.kubevirt.io/v1alpha1
      kind: VirtualMachineSnapshot
      metadata:
        creationTimestamp: "2024-09-18T05:50:13Z"
        finalizers:
        - snapshot.kubevirt.io/vmsnapshot-protection
        generation: 4
        name: snapshot-aquamarine-hare-33
        namespace: homelab
        ownerReferences:
        - apiVersion: kubevirt.io/v1
          blockOwnerDeletion: false
          kind: VirtualMachine
          name: rhel-8
          uid: 1bd72695-04e8-450e-9998-46b62dd92895
        resourceVersion: "1011220"
        uid: c349c24a-7d0a-4db2-b57d-84c947388328
      spec:
        source:
          apiGroup: kubevirt.io
          kind: VirtualMachine
          name: rhel-8
      status:
        conditions:
        - lastProbeTime: null
          lastTransitionTime: "2024-09-18T05:55:13Z"
          reason: snapshot deadline exceeded
          status: "False"
          type: Progressing
        - lastProbeTime: null
          lastTransitionTime: "2024-09-18T05:50:14Z"
          reason: Not ready
          status: "False"
          type: Ready
        - lastProbeTime: null
          lastTransitionTime: "2024-09-18T05:55:13Z"
          reason: snapshot deadline exceeded
          status: "True"
          type: Failure
        indications:
        - Online
        - GuestAgent
        phase: Failed
        readyToUse: false
        sourceUID: 1bd72695-04e8-450e-9998-46b62dd92895
      

      Version-Release number of selected component (if applicable):

      4.16.2
      

      How reproducible:

      100%
      

      Steps to Reproduce:

      Inside the Guest:
      $ mount -o bind /var/ /mnt/test/
      $ chcon -t rpm_var_lib_t /mnt/test
      $ mount -o remount,ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota,suid /mnt/test/
      eclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
      

      Actual results:

      "Deadline exceeded", doesn't say where we were stuck at when the deadline was reached.
      

      Expected results:

      Proper error propagation, tell the user there was a problem freezing the guest fs
      

              skagan@redhat.com Shelly Kagan
              rhn-support-gveitmic Germano Veit Michel
              Natalie Gavrielov Natalie Gavrielov
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: