Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: CNV v4.18.z
Affects Version/s: CNV v4.15.0
Component/s: Storage Ecosystem
Labels:
None

Activity Type:
Quality / Stability / Reliability
Story Points:
0.42
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Epic Link:
CNV-38281
Market:

Regression:
No

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

PX Impact Score:

Description of problem:

We create a VM,
Create VirtualMachineSnapshot snapshot-1, 
Write to VM,
Create VirtualMachineSnapshot snapshot-2,
Write to VM
Create VirtualMachineSnapshot snapshot-3,
Restore snapshot-3, delete the VirtualMachineRestore object 
Restore snapshot-2, delete the VirtualMachineRestore object 
Restore snapshot-1, delete the VirtualMachineRestore object 
Delete VirtualMachineSnapshot snapshot-1, 
Delete VirtualMachineSnapshot snapshot-2, 
Delete VirtualMachineSnapshot snapshot-3, 
Delete VM

Result: 
Two VolumeSnapshots are not cleanedup, we can't delete them, but VirtualMachineSnapshots was deleted succesfully.
Two VolumeSnapshotContents are not cleanedup, and we can't delete them, but VirtualMachineSnapshotContent was deleted succesfully.

Version-Release number of selected component (if applicable):

4.15

How reproducible:

Always

Steps to Reproduce:
Create a VM

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-cnv-6528-1702818904-416998
  namespace: snapshots-test-trident
spec:
  dataVolumeTemplates:
  - metadata:
      name: dv-vm-cnv-6528
      namespace: snapshots-test-trident
    spec:
      contentType: kubevirt
      source:
        http:
          url: <cirros-0.4.0-x86_64-disk.qcow2>
      storage:
        resources:
          requests:
            storage: 1Gi
        storageClassName: trident-csi-fsx
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/domain: vm-cnv-6528-1702818904-416998
        kubevirt.io/vm: vm-cnv-6528-1702818904-416998
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: dv-disk
          rng: {}
        resources:
          requests:
            memory: 64M
      volumes:
      - dataVolume:
          name: dv-vm-cnv-6528
        name: dv-disk

Wait for VM to be Running, Ready
Connect to the console, and create a file inside the VM.
Create VirtualMachineSnapshot number-1,
Wait for VirtualMachineSnapshot status to be readyToUse: True

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
  name: snapshot-vm-cnv-6528-1702818904-416998-number-1
  namespace: snapshots-test-trident
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998

Connect to the console, and create another file inside the VM.
Create VirtualMachineSnapshot number-2,
Wait for VirtualMachineSnapshot status to be readyToUse: True

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
  name: snapshot-vm-cnv-6528-1702818904-416998-number-2
  namespace: snapshots-test-trident
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998

Connect to the console, and create another file inside the VM.
Create VirtualMachineSnapshot number-3,
Wait for VirtualMachineSnapshot status to be readyToUse: True

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
  name: snapshot-vm-cnv-6528-1702818904-416998-number-3
  namespace: snapshots-test-trident
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998

[cloud-user@ocp-psi-executor jenia]$ oc get VirtualMachineSnapshot -n snapshots-test-trident
NAME                                              SOURCEKIND       SOURCENAME                      PHASE       READYTOUSE   CREATIONTIME   ERROR
snapshot-vm-cnv-6528-1702818904-416998-number-1   VirtualMachine   vm-cnv-6528-1702818904-416998   Succeeded   true         7m38s          
snapshot-vm-cnv-6528-1702818904-416998-number-2   VirtualMachine   vm-cnv-6528-1702818904-416998   Succeeded   true         5m35s          
snapshot-vm-cnv-6528-1702818904-416998-number-3   VirtualMachine   vm-cnv-6528-1702818904-416998   Succeeded   true         3m33s

[cloud-user@ocp-psi-executor jenia]$ oc get VolumeSnapshot -n snapshots-test-trident
NAME                                                             READYTOUSE   SOURCEPVC        SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                    CREATIONTIME   AGE
vmsnapshot-a142d868-a08c-4624-b7f4-763563879ef8-volume-dv-disk   true         dv-vm-cnv-6528                           29316Ki       csi-snapclass   snapcontent-2cc5048b-46b8-4b5d-a33d-6ad533a121d9   5m39s          5m39s
vmsnapshot-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d-volume-dv-disk   true         dv-vm-cnv-6528                           29692Ki       csi-snapclass   snapcontent-4359af4b-9f0d-4c3c-889e-2c1c90c82460   3m36s          3m36s
vmsnapshot-fc3591d1-4e7d-4dd4-afc8-4526987e678f-volume-dv-disk   true         dv-vm-cnv-6528                           28688Ki       csi-snapclass   snapcontent-892ee0ba-b311-4958-b03c-815dfb16b751   7m42s          7m42s

Create VirtualMachineRestore for the snapshot number-3

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
  name: restore-snapshot-2
  namespace: snapshots-test-trident
spec:
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
  virtualMachineSnapshotName: snapshot-vm-cnv-6528-1702818904-416998-number-3

Wait Restore Done,
Start the VM,
Wait for VirtualMachine vm-cnv-6528-1702818904-416998 status to be Running, Ready
Stop the VM
Delete the VirtualMachineRestore

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
  creationTimestamp: '2023-12-17T13:21:36Z'
  generation: 7
  managedFields:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:created-by-dynamic-class-creator: {}
      f:spec:
        .: {}
        f:target:
          .: {}
          f:apiGroup: {}
          f:kind: {}
          f:name: {}
        f:virtualMachineSnapshotName: {}
    manager: OpenAPI-Generator
    operation: Update
    time: '2023-12-17T13:21:36Z'
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .: {}
          k:{"uid":"9bb53c3f-45fa-4a16-950c-531077015fb4"}: {}
      f:status:
        .: {}
        f:complete: {}
        f:conditions: {}
        f:deletedDataVolumes: {}
        f:restoreTime: {}
        f:restores: {}
    manager: virt-controller
    operation: Update
    time: '2023-12-17T13:21:38Z'
  name: restore-snapshot-2
  namespace: snapshots-test-trident
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
    uid: 9bb53c3f-45fa-4a16-950c-531077015fb4
  resourceVersion: '295553'
  uid: 8ddd0625-894d-4136-8893-324224730086
spec:
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
  virtualMachineSnapshotName: snapshot-vm-cnv-6528-1702818904-416998-number-3
status:
  complete: true
  conditions:
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:21:38Z'
    reason: Operation complete
    status: 'False'
    type: Progressing
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:21:38Z'
    reason: Operation complete
    status: 'True'
    type: Ready
  deletedDataVolumes:
  - dv-vm-cnv-6528
  restoreTime: '2023-12-17T13:21:38Z'
  restores:
  - dataVolumeName: restore-8ddd0625-894d-4136-8893-324224730086-dv-disk
    persistentVolumeClaim: restore-8ddd0625-894d-4136-8893-324224730086-dv-disk
    volumeName: dv-disk
    volumeSnapshotName: vmsnapshot-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d-volume-dv-disk

Create VirtualMachineRestore for the snapshot number-2

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
  name: restore-snapshot-1
  namespace: snapshots-test-trident
spec:
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
  virtualMachineSnapshotName: snapshot-vm-cnv-6528-1702818904-416998-number-2

Wait Restore Done,
Start the VM,
Wait for VirtualMachine vm-cnv-6528-1702818904-416998 status to be Running, Ready
Stop the VM
Delete the VirtualMachineRestore

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
  creationTimestamp: '2023-12-17T13:22:39Z'
  generation: 7
  managedFields:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:created-by-dynamic-class-creator: {}
      f:spec:
        .: {}
        f:target:
          .: {}
          f:apiGroup: {}
          f:kind: {}
          f:name: {}
        f:virtualMachineSnapshotName: {}
    manager: OpenAPI-Generator
    operation: Update
    time: '2023-12-17T13:22:39Z'
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .: {}
          k:{"uid":"9bb53c3f-45fa-4a16-950c-531077015fb4"}: {}
      f:status:
        .: {}
        f:complete: {}
        f:conditions: {}
        f:deletedDataVolumes: {}
        f:restoreTime: {}
        f:restores: {}
    manager: virt-controller
    operation: Update
    time: '2023-12-17T13:22:40Z'
  name: restore-snapshot-1
  namespace: snapshots-test-trident
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
    uid: 9bb53c3f-45fa-4a16-950c-531077015fb4
  resourceVersion: '296403'
  uid: cf37d697-bed8-4799-a48d-b9e3d5f9e1b3
spec:
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
  virtualMachineSnapshotName: snapshot-vm-cnv-6528-1702818904-416998-number-2
status:
  complete: true
  conditions:
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:22:40Z'
    reason: Operation complete
    status: 'False'
    type: Progressing
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:22:40Z'
    reason: Operation complete
    status: 'True'
    type: Ready
  deletedDataVolumes:
  - restore-8ddd0625-894d-4136-8893-324224730086-dv-disk
  restoreTime: '2023-12-17T13:22:40Z'
  restores:
  - dataVolumeName: restore-cf37d697-bed8-4799-a48d-b9e3d5f9e1b3-dv-disk
    persistentVolumeClaim: restore-cf37d697-bed8-4799-a48d-b9e3d5f9e1b3-dv-disk
    volumeName: dv-disk
    volumeSnapshotName: vmsnapshot-a142d868-a08c-4624-b7f4-763563879ef8-volume-dv-disk

Create VirtualMachineRestore for the snapshot number-1

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
  name: restore-snapshot-0
  namespace: snapshots-test-trident
spec:
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
  virtualMachineSnapshotName: snapshot-vm-cnv-6528-1702818904-416998-number-1

Wait Restore Done,
Start the VM,
Wait for VirtualMachine vm-cnv-6528-1702818904-416998 status to be Running, Ready
Stop the VM
Delete the VirtualMachineRestore

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
  creationTimestamp: '2023-12-17T13:23:40Z'
  generation: 6
  managedFields:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:created-by-dynamic-class-creator: {}
      f:spec:
        .: {}
        f:target:
          .: {}
          f:apiGroup: {}
          f:kind: {}
          f:name: {}
        f:virtualMachineSnapshotName: {}
    manager: OpenAPI-Generator
    operation: Update
    time: '2023-12-17T13:23:40Z'
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .: {}
          k:{"uid":"9bb53c3f-45fa-4a16-950c-531077015fb4"}: {}
      f:status:
        .: {}
        f:complete: {}
        f:conditions: {}
        f:deletedDataVolumes: {}
        f:restoreTime: {}
        f:restores: {}
    manager: virt-controller
    operation: Update
    time: '2023-12-17T13:23:41Z'
  name: restore-snapshot-0
  namespace: snapshots-test-trident
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
    uid: 9bb53c3f-45fa-4a16-950c-531077015fb4
  resourceVersion: '297191'
  uid: f224e556-1727-4263-8734-87e916a14664
spec:
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
  virtualMachineSnapshotName: snapshot-vm-cnv-6528-1702818904-416998-number-1
status:
  complete: true
  conditions:
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:23:41Z'
    reason: Operation complete
    status: 'False'
    type: Progressing
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:23:41Z'
    reason: Operation complete
    status: 'True'
    type: Ready
  deletedDataVolumes:
  - restore-cf37d697-bed8-4799-a48d-b9e3d5f9e1b3-dv-disk
  restoreTime: '2023-12-17T13:23:41Z'
  restores:
  - dataVolumeName: restore-f224e556-1727-4263-8734-87e916a14664-dv-disk
    persistentVolumeClaim: restore-f224e556-1727-4263-8734-87e916a14664-dv-disk
    volumeName: dv-disk
    volumeSnapshotName: vmsnapshot-fc3591d1-4e7d-4dd4-afc8-4526987e678f-volume-dv-disk

Delete VirtualMachineSnapshot number-1

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
  creationTimestamp: '2023-12-17T13:16:43Z'
  finalizers:
  - snapshot.kubevirt.io/vmsnapshot-protection
  generation: 5
  managedFields:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:created-by-dynamic-class-creator: {}
      f:spec:
        .: {}
        f:source:
          .: {}
          f:apiGroup: {}
          f:kind: {}
          f:name: {}
    manager: OpenAPI-Generator
    operation: Update
    time: '2023-12-17T13:16:43Z'
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"snapshot.kubevirt.io/vmsnapshot-protection": {}
      f:status:
        .: {}
        f:conditions: {}
        f:creationTime: {}
        f:indications:
          .: {}
          v:"NoGuestAgent": {}
          v:"Online": {}
        f:phase: {}
        f:readyToUse: {}
        f:snapshotVolumes:
          .: {}
          f:includedVolumes:
            .: {}
            v:"dv-disk": {}
        f:sourceUID: {}
        f:virtualMachineSnapshotContentName: {}
    manager: virt-controller
    operation: Update
    time: '2023-12-17T13:16:44Z'
  name: snapshot-vm-cnv-6528-1702818904-416998-number-1
  namespace: snapshots-test-trident
  resourceVersion: '291697'
  uid: fc3591d1-4e7d-4dd4-afc8-4526987e678f
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:16:44Z'
    reason: Operation complete
    status: 'False'
    type: Progressing
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:16:44Z'
    reason: Operation complete
    status: 'True'
    type: Ready
  creationTime: '2023-12-17T13:16:44Z'
  indications:
  - Online
  - NoGuestAgent
  phase: Succeeded
  readyToUse: true
  snapshotVolumes:
    includedVolumes:
    - dv-disk
  sourceUID: 9bb53c3f-45fa-4a16-950c-531077015fb4
  virtualMachineSnapshotContentName: vmsnapshot-content-fc3591d1-4e7d-4dd4-afc8-4526987e678f

Delete VirtualMachineSnapshot number-2

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
  creationTimestamp: '2023-12-17T13:18:46Z'
  finalizers:
  - snapshot.kubevirt.io/vmsnapshot-protection
  generation: 5
  managedFields:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:created-by-dynamic-class-creator: {}
      f:spec:
        .: {}
        f:source:
          .: {}
          f:apiGroup: {}
          f:kind: {}
          f:name: {}
    manager: OpenAPI-Generator
    operation: Update
    time: '2023-12-17T13:18:46Z'
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"snapshot.kubevirt.io/vmsnapshot-protection": {}
      f:status:
        .: {}
        f:conditions: {}
        f:creationTime: {}
        f:indications:
          .: {}
          v:"NoGuestAgent": {}
          v:"Online": {}
        f:phase: {}
        f:readyToUse: {}
        f:snapshotVolumes:
          .: {}
          f:includedVolumes:
            .: {}
            v:"dv-disk": {}
        f:sourceUID: {}
        f:virtualMachineSnapshotContentName: {}
    manager: virt-controller
    operation: Update
    time: '2023-12-17T13:18:47Z'
  name: snapshot-vm-cnv-6528-1702818904-416998-number-2
  namespace: snapshots-test-trident
  resourceVersion: '293310'
  uid: a142d868-a08c-4624-b7f4-763563879ef8
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:18:47Z'
    reason: Operation complete
    status: 'False'
    type: Progressing
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:18:47Z'
    reason: Operation complete
    status: 'True'
    type: Ready
  creationTime: '2023-12-17T13:18:47Z'
  indications:
  - Online
  - NoGuestAgent
  phase: Succeeded
  readyToUse: true
  snapshotVolumes:
    includedVolumes:
    - dv-disk
  sourceUID: 9bb53c3f-45fa-4a16-950c-531077015fb4
  virtualMachineSnapshotContentName: vmsnapshot-content-a142d868-a08c-4624-b7f4-763563879ef8

Delete VirtualMachineSnapshot number-3

apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
  creationTimestamp: '2023-12-17T13:20:48Z'
  finalizers:
  - snapshot.kubevirt.io/vmsnapshot-protection
  generation: 5
  managedFields:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:created-by-dynamic-class-creator: {}
      f:spec:
        .: {}
        f:source:
          .: {}
          f:apiGroup: {}
          f:kind: {}
          f:name: {}
    manager: OpenAPI-Generator
    operation: Update
    time: '2023-12-17T13:20:48Z'
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"snapshot.kubevirt.io/vmsnapshot-protection": {}
      f:status:
        .: {}
        f:conditions: {}
        f:creationTime: {}
        f:indications:
          .: {}
          v:"NoGuestAgent": {}
          v:"Online": {}
        f:phase: {}
        f:readyToUse: {}
        f:snapshotVolumes:
          .: {}
          f:includedVolumes:
            .: {}
            v:"dv-disk": {}
        f:sourceUID: {}
        f:virtualMachineSnapshotContentName: {}
    manager: virt-controller
    operation: Update
    time: '2023-12-17T13:20:49Z'
  name: snapshot-vm-cnv-6528-1702818904-416998-number-3
  namespace: snapshots-test-trident
  resourceVersion: '294916'
  uid: b4ecbd71-5324-4e11-ad57-6c76cba5eb6d
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cnv-6528-1702818904-416998
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:20:49Z'
    reason: Operation complete
    status: 'False'
    type: Progressing
  - lastProbeTime: null
    lastTransitionTime: '2023-12-17T13:20:49Z'
    reason: Operation complete
    status: 'True'
    type: Ready
  creationTime: '2023-12-17T13:20:49Z'
  indications:
  - Online
  - NoGuestAgent
  phase: Succeeded
  readyToUse: true
  snapshotVolumes:
    includedVolumes:
    - dv-disk
  sourceUID: 9bb53c3f-45fa-4a16-950c-531077015fb4
  virtualMachineSnapshotContentName: vmsnapshot-content-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d

Delete VirtualMachine vm-cnv-6528-1702818904-416998

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kubemacpool.io/transaction-timestamp: '2023-12-17T13:24:33.877290924Z'
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1
    restore.kubevirt.io/lastRestoreUID: restore-snapshot-0-f224e556-1727-4263-8734-87e916a14664
  creationTimestamp: '2023-12-17T13:15:04Z'
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  generation: 22
  managedFields:
  - apiVersion: kubevirt.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:created-by-dynamic-class-creator: {}
      f:spec:
        .: {}
        f:template:
          .: {}
          f:metadata:
            .: {}
            f:labels:
              .: {}
              f:kubevirt.io/domain: {}
              f:kubevirt.io/vm: {}
          f:spec:
            .: {}
            f:domain:
              .: {}
              f:devices:
                .: {}
                f:disks: {}
                f:rng: {}
              f:resources:
                .: {}
                f:requests:
                  .: {}
                  f:memory: {}
    manager: OpenAPI-Generator
    operation: Update
    time: '2023-12-17T13:15:04Z'
  - apiVersion: kubevirt.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubevirt.io/latest-observed-api-version: {}
          f:kubevirt.io/storage-observed-api-version: {}
          f:restore.kubevirt.io/lastRestoreUID: {}
        f:finalizers:
          .: {}
          v:"kubevirt.io/virtualMachineControllerFinalize": {}
      f:spec:
        f:dataVolumeTemplates: {}
        f:running: {}
        f:template:
          f:spec:
            f:volumes: {}
    manager: Go-http-client
    operation: Update
    time: '2023-12-17T13:24:33Z'
  - apiVersion: kubevirt.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:conditions: {}
        f:desiredGeneration: {}
        f:observedGeneration: {}
        f:printableStatus: {}
        f:volumeSnapshotStatuses: {}
    manager: Go-http-client
    operation: Update
    subresource: status
    time: '2023-12-17T13:24:39Z'
  name: vm-cnv-6528-1702818904-416998
  namespace: snapshots-test-trident
  resourceVersion: '297912'
  uid: 9bb53c3f-45fa-4a16-950c-531077015fb4
spec:
  dataVolumeTemplates:
  - metadata:
      creationTimestamp: null
      labels:
        created-by-dynamic-class-creator: 'Yes'
      name: restore-f224e556-1727-4263-8734-87e916a14664-dv-disk
      namespace: snapshots-test-trident
    spec:
      contentType: kubevirt
      source:
        http:
          url: <cirros-0.4.0-x86_64-disk.qcow2>
      storage:
        resources:
          requests:
            storage: 1Gi
        storageClassName: trident-csi-fsx
  running: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: vm-cnv-6528-1702818904-416998
        kubevirt.io/vm: vm-cnv-6528-1702818904-416998
    spec:
      architecture: amd64
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: dv-disk
          rng: {}
        machine:
          type: pc-q35-rhel9.2.0
        resources:
          requests:
            memory: 64M
      volumes:
      - dataVolume:
          name: restore-f224e556-1727-4263-8734-87e916a14664-dv-disk
        name: dv-disk
status:
  conditions:
  - lastProbeTime: '2023-12-17T13:24:39Z'
    lastTransitionTime: '2023-12-17T13:24:39Z'
    message: VMI does not exist
    reason: VMINotExists
    status: 'False'
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: null
    status: 'True'
    type: LiveMigratable
  desiredGeneration: 22
  observedGeneration: 22
  printableStatus: Stopped
  volumeSnapshotStatuses:
  - enabled: true
    name: dv-disk

Actual results:
VolumeSnapshots and VolumeSnapshotContent are not cleaned up

$ oc get vm -n snapshots-test-trident
No resources found in snapshots-test-trident namespace.
$ oc get dv -n snapshots-test-trident
No resources found in snapshots-test-trident namespace.
$ oc get pvc -n snapshots-test-trident
No resources found in snapshots-test-trident namespace.

$ oc get VirtualMachineSnapshot -n snapshots-test-trident
No resources found in snapshots-test-trident namespace.

$ oc get VolumeSnapshot -n snapshots-test-trident
NAME                                                             READYTOUSE   SOURCEPVC        SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                    CREATIONTIME   AGE
vmsnapshot-a142d868-a08c-4624-b7f4-763563879ef8-volume-dv-disk   true         dv-vm-cnv-6528                           29316Ki       csi-snapclass   snapcontent-2cc5048b-46b8-4b5d-a33d-6ad533a121d9   16m            16m
vmsnapshot-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d-volume-dv-disk   true         dv-vm-cnv-6528                           29692Ki       csi-snapclass   snapcontent-4359af4b-9f0d-4c3c-889e-2c1c90c82460   14m            14m

$ oc get VirtualMachineSnapshotContent -n snapshots-test-trident
No resources found in snapshots-test-trident namespace.

$ oc get VolumeSnapshotContent | grep snapshots-test-trident
snapcontent-2cc5048b-46b8-4b5d-a33d-6ad533a121d9   true         30019584      Delete           csi.trident.netapp.io   csi-snapclass         vmsnapshot-a142d868-a08c-4624-b7f4-763563879ef8-volume-dv-disk    snapshots-test-trident     17m
snapcontent-4359af4b-9f0d-4c3c-889e-2c1c90c82460   true         30404608      Delete           csi.trident.netapp.io   csi-snapclass         vmsnapshot-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d-volume-dv-disk    snapshots-test-trident     15m

$ oc get VolumeSnapshot -n snapshots-test-trident vmsnapshot-a142d868-a08c-4624-b7f4-763563879ef8-volume-dv-disk -oyaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  creationTimestamp: "2023-12-17T13:18:46Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-12-17T13:24:40Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  generation: 2
  labels:
    snapshot.kubevirt.io/source-vm-name: vm-cnv-6528-1702818904-416998
    snapshot.kubevirt.io/source-vm-namespace: snapshots-test-trident
  name: vmsnapshot-a142d868-a08c-4624-b7f4-763563879ef8-volume-dv-disk
  namespace: snapshots-test-trident
  ownerReferences:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachineSnapshotContent
    name: vmsnapshot-content-a142d868-a08c-4624-b7f4-763563879ef8
    uid: abc2cb24-bf83-454d-85f1-e5083f1c1a2c
  resourceVersion: "297949"
  uid: 2cc5048b-46b8-4b5d-a33d-6ad533a121d9
spec:
  source:
    persistentVolumeClaimName: dv-vm-cnv-6528
  volumeSnapshotClassName: csi-snapclass
status:
  boundVolumeSnapshotContentName: snapcontent-2cc5048b-46b8-4b5d-a33d-6ad533a121d9
  creationTime: "2023-12-17T13:18:46Z"
  readyToUse: true
  restoreSize: 29316Ki

$ oc get VolumeSnapshot -n snapshots-test-trident vmsnapshot-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d-volume-dv-disk -oyaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  creationTimestamp: "2023-12-17T13:20:49Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-12-17T13:24:41Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  generation: 2
  labels:
    snapshot.kubevirt.io/source-vm-name: vm-cnv-6528-1702818904-416998
    snapshot.kubevirt.io/source-vm-namespace: snapshots-test-trident
  name: vmsnapshot-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d-volume-dv-disk
  namespace: snapshots-test-trident
  ownerReferences:
  - apiVersion: snapshot.kubevirt.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachineSnapshotContent
    name: vmsnapshot-content-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d
    uid: c1b22e25-f365-487c-962b-c0c654845d91
  resourceVersion: "297974"
  uid: 4359af4b-9f0d-4c3c-889e-2c1c90c82460
spec:
  source:
    persistentVolumeClaimName: dv-vm-cnv-6528
  volumeSnapshotClassName: csi-snapclass
status:
  boundVolumeSnapshotContentName: snapcontent-4359af4b-9f0d-4c3c-889e-2c1c90c82460
  creationTime: "2023-12-17T13:20:49Z"
  readyToUse: true
  restoreSize: 29692Ki

$ oc get VolumeSnapshotContent snapcontent-2cc5048b-46b8-4b5d-a33d-6ad533a121d9 -oyaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  annotations:
    snapshot.storage.kubernetes.io/volumesnapshot-being-deleted: "yes"
  creationTimestamp: "2023-12-17T13:18:46Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-12-17T13:24:40Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
  generation: 2
  name: snapcontent-2cc5048b-46b8-4b5d-a33d-6ad533a121d9
  resourceVersion: "297947"
  uid: 53b32f06-0c35-46bd-b98c-5c85a646a921
spec:
  deletionPolicy: Delete
  driver: csi.trident.netapp.io
  source:
    volumeHandle: pvc-2e553084-a0c8-4ead-a600-c5b66d30ea73
  volumeSnapshotClassName: csi-snapclass
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    name: vmsnapshot-a142d868-a08c-4624-b7f4-763563879ef8-volume-dv-disk
    namespace: snapshots-test-trident
    resourceVersion: "293278"
    uid: 2cc5048b-46b8-4b5d-a33d-6ad533a121d9
status:
  creationTime: 1702819126000000000
  readyToUse: true
  restoreSize: 30019584
  snapshotHandle: pvc-2e553084-a0c8-4ead-a600-c5b66d30ea73/snapshot-2cc5048b-46b8-4b5d-a33d-6ad533a121d9

$ oc get VolumeSnapshotContent snapcontent-4359af4b-9f0d-4c3c-889e-2c1c90c82460 -oyaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  annotations:
    snapshot.storage.kubernetes.io/volumesnapshot-being-deleted: "yes"
  creationTimestamp: "2023-12-17T13:20:49Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-12-17T13:24:41Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
  generation: 2
  name: snapcontent-4359af4b-9f0d-4c3c-889e-2c1c90c82460
  resourceVersion: "297973"
  uid: f7695fcd-18de-4d8e-b596-cc25a24c9ad8
spec:
  deletionPolicy: Delete
  driver: csi.trident.netapp.io
  source:
    volumeHandle: pvc-2e553084-a0c8-4ead-a600-c5b66d30ea73
  volumeSnapshotClassName: csi-snapclass
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    name: vmsnapshot-b4ecbd71-5324-4e11-ad57-6c76cba5eb6d-volume-dv-disk
    namespace: snapshots-test-trident
    resourceVersion: "294880"
    uid: 4359af4b-9f0d-4c3c-889e-2c1c90c82460
status:
  creationTime: 1702819249000000000
  readyToUse: true
  restoreSize: 30404608
  snapshotHandle: pvc-2e553084-a0c8-4ead-a600-c5b66d30ea73/snapshot-4359af4b-9f0d-4c3c-889e-2c1c90c82460

Expected results:

All cleaned up, no VolumeSnapshots or VolumeSnapshotContent objects left

Additional info:
ryasharz@redhat.com sees that these volumes are staying on the node, but becoming offline at some point. More info from Rabin:

https://docs.google.com/document/d/1D-XSlp-9_UtYt1a-rRgaAnF3R3w7pooIZczbjGeVY7E/edit

is documented by

CNV-38150 Document workaround for CNV-36433

Closed

links to

NetApp/trident Upstream issue: Trident fail to delete child FlexClone of a FlexClone #878

Assignee:: Alex Kalenyuk

Reporter:: Jenia Peimer

QA Contact:: Natalie Gavrielov

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Created:: 2023/12/17 3:06 PM

Updated:: 2025/08/25 1:17 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates