Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-43123

[4.16] Downloading virtctl memory-dump results in empty tar file.

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • CNV v4.16.0
    • None
    • Storage Platform
    • None
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • CNV v4.16.0.rhel9-2730
    • ---
    • ---
    • Storage Core Sprint 255
    • Important
    • No

      Description of problem:

      Attempting to download a Memory Dump from a VM results in an empty tar directory. 

      Version-Release number of selected component (if applicable):

      4.12 and 4.13

      How reproducible:

      Steps to Reproduce:

      On 4.12:
      
      Create dump with: 
      #virtctl memory-dump get rhel8-feqarbepxe4sgxcf --claim-name=shaggy-dump --create-claim
      
      
      $oc describe vm rhel8-feqarbepxe4sgxcf 
         Memory Dump Request:
          Claim Name:       shaggy-dump
          End Timestamp:    2024-03-07T18:56:21Z
          File Name:        rhel8-feqarbepxe4sgxcf-shaggy-dump-20240307-185618.memory.dump
          Phase:            Completed
          Start Timestamp:  2024-03-07T18:56:18Z
      
      $oc get pvc
      NAME                     STATUS   VOLUME                                     CAPACITY      ACCESS MODES   STORAGECLASS   AGE
      shaggy-dump              Bound    pvc-5c2b9e8e-cab3-42f0-b00d-06332072e5cb   2267Mi        RWO            nfs-client     35s
      
      
      Download it with: 
      #virtctl vmexport download shaggy-dump --pvc=shaggy-dump --volume=pvc-5c2b9e8e-cab3-42f0-b00d-06332072e5cb --output=shaggy.img.gz
      
      However it looks to be empty? 
      [kni@cee ~]$ tar -ztvf shaggy.img.gz
      drwxrwxrwx root/root         0 2024-03-07 13:56 ./
      
      
      

       

      On 4.13 using the all in one command:
      
      $ virtctl memory-dump get rhel8-dumptest --create-claim --claim-name=dumptest --output=dumptest.tar.gz
      PVC shaggy/dumptest created
      Successfully submitted memory dump request of VM rhel8-dumptest
      Waiting for memorydump  to complete, current phase: Associating...
      Waiting for memorydump  to complete, current phase: InProgress...
      Waiting for memorydump  to complete, current phase: InProgress...
      Memory dump completed successfully
      VirtualMachineExport 'shaggy/export-rhel8-dumptest-dumptest' created succesfully
      Downloading file: 122 B [___________________] ? p/s                                                                                                                                                
      Download finished succesfully
      VirtualMachineExport 'shaggy/export-rhel8-dumptest-dumptest' deleted succesfully
      
      $ tar -ztvf dumptest.tar.gz 
      drwxrwxrwx virt-exportserver/107 0 2024-03-07 14:15 ./
      
      Above downloaded dump is empty also
      
      
      $oc describe vm rhel8-dumptest
      
        Memory Dump Request:
          Claim Name:       dumptest
          End Timestamp:    2024-03-07T19:15:36Z
          File Name:        rhel8-dumptest-dumptest-20240307-191534.memory.dump
          Phase:            Completed
          Start Timestamp:  2024-03-07T19:15:34Z
      
      
      Attach dumptest pvc to vm in the GUI, stop VM, start VM 
      
      [kni@cee ~]$ oc get pods -o wide
      NAME                                 READY   STATUS    RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
      hp-volume-9dkqd                      1/1     Running   0          7s    10.128.3.25   openshift-worker-1   <none>           <none>
      virt-launcher-rhel8-dumptest-rs5pp   1/1     Running   0          10s   10.128.3.24   openshift-worker-1   <none>           1/1
      
      
      oc debug node/openshift-worker-1 
      we see the dump pvc mounted here on the host: 
      
      sh-4.4# df | grep dumptest
      192.168.111.1:/shaggy-rhel8-dumptest-pvc-0a340671-13fd-4e2e-b2ab-8b45d3bbf8a2 3669661696 326250496 3343411200   9% /host/var/lib/kubelet/pods/0d24eb5d-9f66-4262-99c1-d75b098fa144/volumes/kubernetes.io~nfs/pvc-0a340671-13fd-4e2e-b2ab-8b45d3bbf8a2
      192.168.111.1:/shaggy-dumptest-pvc-d5a7ac30-5c00-4ab4-870c-b72effdce7e7       3669661696 326250496 3343411200   9% /host/var/lib/kubelet/pods/4434f45e-b5e7-47af-a148-a23ef5520edf/volumes/kubernetes.io~nfs/pvc-d5a7ac30-5c00-4ab4-870c-b72effdce7e7    
      
      
      sh-4.4# cd /host/var/lib/kubelet/pods/4434f45e-b5e7-47af-a148-a23ef5520edf/volumes/kubernetes.io~nfs/pvc-d5a7ac30-5c00-4ab4-870c-b72effdce7e7
      sh-4.4# ls -la
      total 2113732
      drwxrwxrwx. 2  107  107         65 Mar  7 19:15 .
      drwxr-x---. 3 root root         54 Mar  7 19:17 ..
      -rw-------. 1  107  107 2164458827 Mar  7 19:15 rhel8-dumptest-dumptest-20240307-191534.memory.dump
      
      Here we see the file that we would need for the dump, so we know the dump was created successfully
      
      

       

      Actual results:

      Downloaded tar file is empty and only 122B

      Expected results:

      The tar file should contain the entire memory image and be approx 2GB to match the amount of memory 

            [CNV-43123] [4.16] Downloading virtctl memory-dump results in empty tar file.

            Kevin Alon Goldblatt added a comment - - edited

            Tested using code:
            v4.16.0.rhel9-2746

            oc version
            Client Version: 4.16.0-rc.5
            Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
            Server Version: 4.16.0-rc.5
            Kubernetes Version: v1.29.5+b2f5b11
            [cloud-user@ocp-psi-executor-xl ~]$ oc get csv -n openshift-cnv
            NAME                                       DISPLAY                    VERSION   REPLACES                                   PHASE
            kubevirt-hyperconverged-operator.v4.16.0   OpenShift Virtualization   4.16.0    kubevirt-hyperconverged-operator.v4.15.2   Succeeded

            Tested using the scenario in the description:

            Tested 10 times on storage classes below and the download worked fine every time:
            ocs-storagecluster-ceph-rbd

            ocs-storagecluster-ceph-rbd-virtualization

            ocs-storagecluster-cephfs

            Eg.
            With ocs-storagecluster-ceph-rbd:
            tar -ztvf  vm-43123-dump.img.gz
            drwxrwsr-x 107/107           0 2024-07-10 10:41 ./
            drwxrws--- root/107          0 2024-07-10 10:41 ./lost+found/
            rw-rw--- 107/107  2147288563 2024-07-10 10:41 ./vm-43123-vm-43123-dump-20240710-144117.memory.dump

            Kevin Alon Goldblatt added a comment - - edited Tested using code: v4.16.0.rhel9-2746 oc version Client Version: 4.16.0-rc.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: 4.16.0-rc.5 Kubernetes Version: v1.29.5+b2f5b11 [cloud-user@ocp-psi-executor-xl ~] $ oc get csv -n openshift-cnv NAME                                       DISPLAY                    VERSION   REPLACES                                   PHASE kubevirt-hyperconverged-operator.v4.16.0   OpenShift Virtualization   4.16.0    kubevirt-hyperconverged-operator.v4.15.2   Succeeded Tested using the scenario in the description: Tested 10 times on storage classes below and the download worked fine every time: ocs-storagecluster-ceph-rbd ocs-storagecluster-ceph-rbd-virtualization ocs-storagecluster-cephfs Eg. With ocs-storagecluster-ceph-rbd: tar -ztvf  vm-43123-dump.img.gz drwxrwsr-x 107/107           0 2024-07-10 10:41 ./ drwxrws--- root/107          0 2024-07-10 10:41 ./lost+found/ rw-rw --- 107/107  2147288563 2024-07-10 10:41 ./vm-43123-vm-43123-dump-20240710-144117.memory.dump

            A build that contains the fix PR(s) for this bug is available, moving status from MODIFIED to ON_QA.
            The first build that contains https://github.com/kubevirt/kubevirt/pull/12156 is CNV v4.16.0.rhel9-2730.

            cnv-qe jira added a comment - A build that contains the fix PR(s) for this bug is available, moving status from MODIFIED to ON_QA. The first build that contains https://github.com/kubevirt/kubevirt/pull/12156 is CNV v4.16.0.rhel9-2730 .

            cnv-qe jira added a comment -

            All linked PR(s) of this bug have been merged, moving status from POST to MODIFIED.

            https://github.com/kubevirt/kubevirt/pull/12156 merged at 2024-06-22 04:53:35

            cnv-qe jira added a comment - All linked PR(s) of this bug have been merged, moving status from POST to MODIFIED. https://github.com/kubevirt/kubevirt/pull/12156 merged at 2024-06-22 04:53:35

            cnv-qe jira added a comment -

            This bug has a linked PR, moving status from ASSIGNED to POST.

            cnv-qe jira added a comment - This bug has a linked PR, moving status from ASSIGNED to POST.

              rh-ee-alromero Alvaro Romero
              shaselde@redhat.com Sean Haselden
              Kevin Alon Goldblatt Kevin Alon Goldblatt
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: