-
Sub-task
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
-
False
-
-
False
-
ToDo
-
-
-
0
-
0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
Description of problem:
- This is simply a clean write up and simple recreate for the bug Amos found in https://issues.redhat.com/browse/OADP-4230
- Environment: OCP 4.16
OADP-1.4 devel install - setup any sample app from https://github.com/openshift/oadp-operator/tree/master/tests/e2e/sample-applications. In this case I used https://github.com/openshift/oadp-operator/blob/master/tests/e2e/sample-applications/minimal-8csivol/minimal-3csivol.yaml
- Ensure the app is running normally
- DPA
Version: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample namespace: openshift-adp spec: backupLocations: - velero: config: profile: default region: us-west-2 credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: cvpbucketuswest2 prefix: velero provider: aws configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - csi - openshift - aws
In the deployment config set the replicaset settings to 0
32 apiVersion: apps/v1 33 kind: Deployment 34 metadata: 35 name: minimal-3csivol 36 namespace: minimal-3csivol 37 spec: 38 replicas: 1 --> 0 There are additional replica settings in status, update those to 0 as well
pps/minimal-3csivol -n minimal-3csivol -o yaml | grep -B 5 -A 5 replica namespace: minimal-3csivol resourceVersion: "6891044" uid: 4bcdd8d0-6788-4ef9-8004-0dd7fe257e86 spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: minimal-3csivol strategy: -- reason: NewReplicaSetAvailable status: "True" type: Progressing observedGeneration: 1 readyReplicas: 1 replicas: 1 updatedReplicas: 1
The number of pods running should move from 1/1 to 0/1
Notice that the pvc's are all still present.
oc get all -n minimal-3csivol Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/minimal-3csivol 0/0 0 0 3m1s NAME DESIRED CURRENT READY AGE replicaset.apps/minimal-3csivol-7b779fbd4d 0 0 0 3m2s
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/oadp-operator$ oc get pvc -n minimal-3csivol NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE volume1 Bound pvc-9fdb9241-86db-4e2f-bb8f-76c4ab2ee763 1Gi RWO gp3-csi 3m49s volume2 Bound pvc-2853b5c5-2767-4f0f-9284-291d1a2b9329 1Gi RWO gp3-csi 3m49s volume3 Bound pvc-4bb128f8-4ced-41dc-99a5-83066597f703 1Gi RWO gp3-csi 3m49s whayutin@thinkdoe:~/OPENSHIFT/git/OADP/oadp-operator$
Execute the backup:
1.
velero backup create no-pod1 --include-namespaces minimal-3csivol --snapshot-volumes=true
pod backup description:
Name: no-pod1 Namespace: openshift-adp Labels: velero.io/storage-location=dpa-sample-1 Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.29.5+87992f4 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=29 Phase: Completed Namespaces: Included: minimal-3csivol Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: auto Label selector: <none> Or label selector: <none> Storage Location: dpa-sample-1 Velero-Native Snapshot PVs: true Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2024-06-05 22:25:08 +0000 UTC Completed: 2024-06-05 22:26:02 +0000 UTC Expiration: 2024-07-05 22:25:08 +0000 UTC Total items to be backed up: 67 Items backed up: 67 Backup Item Operations: Operation for volumesnapshots.snapshot.storage.k8s.io minimal-3csivol/velero-volume1-t5hn4: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: minimal-3csivol/velero-volume1-t5hn4/2024-06-05T22:25:14Z Items to Update: volumesnapshots.snapshot.storage.k8s.io minimal-3csivol/velero-volume1-t5hn4 volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-824ebaba-35b0-4dd9-883d-75393e25940d Phase: Completed Created: 2024-06-05 22:25:14 +0000 UTC Started: 2024-06-05 22:25:14 +0000 UTC Updated: 2024-06-05 22:25:38 +0000 UTC Operation for volumesnapshots.snapshot.storage.k8s.io minimal-3csivol/velero-volume2-rq5s9: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: minimal-3csivol/velero-volume2-rq5s9/2024-06-05T22:25:19Z Items to Update: volumesnapshots.snapshot.storage.k8s.io minimal-3csivol/velero-volume2-rq5s9 volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-36493369-50da-476c-9f91-d068baad3531 Phase: Completed Created: 2024-06-05 22:25:19 +0000 UTC Started: 2024-06-05 22:25:19 +0000 UTC Updated: 2024-06-05 22:25:48 +0000 UTC Operation for volumesnapshots.snapshot.storage.k8s.io minimal-3csivol/velero-volume3-5rb56: Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper Operation ID: minimal-3csivol/velero-volume3-5rb56/2024-06-05T22:25:24Z Items to Update: volumesnapshots.snapshot.storage.k8s.io minimal-3csivol/velero-volume3-5rb56 volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-92af2390-9b2b-45b5-8127-d8e426aa62bb Phase: Completed Created: 2024-06-05 22:25:24 +0000 UTC Started: 2024-06-05 22:25:24 +0000 UTC Updated: 2024-06-05 22:25:58 +0000 UTC Resource List: apps/v1/Deployment: - minimal-3csivol/minimal-3csivol apps/v1/ReplicaSet: - minimal-3csivol/minimal-3csivol-7b779fbd4d authorization.openshift.io/v1/RoleBinding: - minimal-3csivol/system:deployers - minimal-3csivol/system:image-builders - minimal-3csivol/system:image-pullers rbac.authorization.k8s.io/v1/RoleBinding: - minimal-3csivol/system:deployers - minimal-3csivol/system:image-builders - minimal-3csivol/system:image-pullers snapshot.storage.k8s.io/v1/VolumeSnapshot: - minimal-3csivol/velero-volume1-t5hn4 - minimal-3csivol/velero-volume2-rq5s9 - minimal-3csivol/velero-volume3-5rb56 snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - csi-aws-vsc snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-36493369-50da-476c-9f91-d068baad3531 - snapcontent-824ebaba-35b0-4dd9-883d-75393e25940d - snapcontent-92af2390-9b2b-45b5-8127-d8e426aa62bb v1/ConfigMap: - minimal-3csivol/kube-root-ca.crt - minimal-3csivol/openshift-service-ca.crt v1/Event: - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba11d6b2bb5 - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba29cf81646 - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba324780a75 - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba32802416f - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba349a18004 - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba473b91b81 - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba47781be71 - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba4edb8081b - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba512db0186 - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63ba513c9f20d - minimal-3csivol/minimal-3csivol-7b779fbd4d-pjncc.17d63bcdd5de88a8 - minimal-3csivol/minimal-3csivol-7b779fbd4d-plfxx.17d63bcdd89f25d6 - minimal-3csivol/minimal-3csivol-7b779fbd4d-plfxx.17d63bcdf06aeee4 - minimal-3csivol/minimal-3csivol-7b779fbd4d-plfxx.17d63bcdf2097da1 - minimal-3csivol/minimal-3csivol-7b779fbd4d-plfxx.17d63bce1f6edaca - minimal-3csivol/minimal-3csivol-7b779fbd4d-plfxx.17d63bce3d1bb7b7 - minimal-3csivol/minimal-3csivol-7b779fbd4d-plfxx.17d63bce3e07b9af - minimal-3csivol/minimal-3csivol-7b779fbd4d-plfxx.17d63bddfb1d5c4b - minimal-3csivol/minimal-3csivol-7b779fbd4d.17d63ba11da3db54 - minimal-3csivol/minimal-3csivol-7b779fbd4d.17d63bcdd5dfc0d3 - minimal-3csivol/minimal-3csivol-7b779fbd4d.17d63bcdd81123bb - minimal-3csivol/minimal-3csivol-7b779fbd4d.17d63bddfb11fcb2 - minimal-3csivol/minimal-3csivol.17d63ba11b9c3faf - minimal-3csivol/minimal-3csivol.17d63bcdd5e9ea46 - minimal-3csivol/minimal-3csivol.17d63bddfa340e83 - minimal-3csivol/volume1.17d63ba11d5be64b - minimal-3csivol/volume1.17d63ba172060b57 - minimal-3csivol/volume1.17d63ba1720e25b8 - minimal-3csivol/volume1.17d63ba274a90679 - minimal-3csivol/volume2.17d63ba11fc61fa1 - minimal-3csivol/volume2.17d63ba1727f25a6 - minimal-3csivol/volume2.17d63ba172882242 - minimal-3csivol/volume2.17d63ba27496fc2f - minimal-3csivol/volume3.17d63ba122511355 - minimal-3csivol/volume3.17d63ba172eb531c - minimal-3csivol/volume3.17d63ba172eff1da - minimal-3csivol/volume3.17d63ba274b5723d v1/Namespace: - minimal-3csivol v1/PersistentVolume: - pvc-685c140e-e654-4ce1-9d5c-dcfb6d5bb54a - pvc-80e98020-fc44-4326-a80e-ed5890b4c6a8 - pvc-f5ae8bae-df65-4ffe-82c4-b0771445815f v1/PersistentVolumeClaim: - minimal-3csivol/volume1 - minimal-3csivol/volume2 - minimal-3csivol/volume3 v1/Secret: - minimal-3csivol/builder-dockercfg-826rv - minimal-3csivol/default-dockercfg-d22jl - minimal-3csivol/deployer-dockercfg-bl4n5 v1/ServiceAccount: - minimal-3csivol/builder - minimal-3csivol/default - minimal-3csivol/deployer Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: minimal-3csivol/volume1: Snapshot: Operation ID: minimal-3csivol/velero-volume1-t5hn4/2024-06-05T22:25:14Z Snapshot Content Name: snapcontent-824ebaba-35b0-4dd9-883d-75393e25940d Storage Snapshot ID: snap-05256c003a5db0bb3 Snapshot Size (bytes): 1073741824 CSI Driver: ebs.csi.aws.com minimal-3csivol/volume2: Snapshot: Operation ID: minimal-3csivol/velero-volume2-rq5s9/2024-06-05T22:25:19Z Snapshot Content Name: snapcontent-36493369-50da-476c-9f91-d068baad3531 Storage Snapshot ID: snap-0a80ce7b9cb6f513b Snapshot Size (bytes): 1073741824 CSI Driver: ebs.csi.aws.com minimal-3csivol/volume3: Snapshot: Operation ID: minimal-3csivol/velero-volume3-5rb56/2024-06-05T22:25:24Z Snapshot Content Name: snapcontent-92af2390-9b2b-45b5-8127-d8e426aa62bb Storage Snapshot ID: snap-0a99c20e93d52e57d Snapshot Size (bytes): 1073741824 CSI Driver: ebs.csi.aws.com Pod Volume Backups: <none included> HooksAttempted: 0 HooksFailed: 0
The backup completes successfully.
2. Delete the app namespace
oc delete namespace minimal-3csivol
3. Restore the backup
velero restore create no-pod1 --from-backup no-pod1
4. The restore will fail w/:
Errors: Velero: <none> Cluster: <none> Namespaces: minimal-3csivol: fail to patch dynamic PV, err: context deadline exceeded, PVC: volume2, PV: pvc-80e98020-fc44-4326-a80e-ed5890b4c6a8 fail to patch dynamic PV, err: context deadline exceeded, PVC: volume3, PV: pvc-685c140e-e654-4ce1-9d5c-dcfb6d5bb54a fail to patch dynamic PV, err: context deadline exceeded, PVC: volume1, PV: pvc-f5ae8bae-df65-4ffe-82c4-b0771445815f
Full Details of restore:
Name: no-pod1 Namespace: openshift-adp Labels: <none> Annotations: <none> Phase: PartiallyFailed (run 'velero restore logs no-pod1' for more information) Total items to be restored: 30 Items restored: 30 Started: 2024-06-05 22:28:17 +0000 UTC Completed: 2024-06-05 22:38:19 +0000 UTC Warnings: Velero: <none> Cluster: could not restore, VolumeSnapshotContent "snapcontent-36493369-50da-476c-9f91-d068baad3531" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, VolumeSnapshotContent "snapcontent-824ebaba-35b0-4dd9-883d-75393e25940d" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, VolumeSnapshotContent "snapcontent-92af2390-9b2b-45b5-8127-d8e426aa62bb" already exists. Warning: the in-cluster version is different than the backed-up version Namespaces: minimal-3csivol: could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, ConfigMap "openshift-service-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, RoleBinding "system:deployers" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, RoleBinding "system:image-builders" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, RoleBinding "system:image-pullers" already exists. Warning: the in-cluster version is different than the backed-up version Errors: Velero: <none> Cluster: <none> Namespaces: minimal-3csivol: fail to patch dynamic PV, err: context deadline exceeded, PVC: volume2, PV: pvc-80e98020-fc44-4326-a80e-ed5890b4c6a8 fail to patch dynamic PV, err: context deadline exceeded, PVC: volume3, PV: pvc-685c140e-e654-4ce1-9d5c-dcfb6d5bb54a fail to patch dynamic PV, err: context deadline exceeded, PVC: volume1, PV: pvc-f5ae8bae-df65-4ffe-82c4-b0771445815f Backup: no-pod1 Namespaces: Included: all namespaces found in the backup Excluded: <none> Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io Cluster-scoped: auto Namespace mappings: <none> Label selector: <none> Or label selector: <none> Restore PVs: auto CSI Snapshot Restores: <none included> Existing Resource Policy: <none> ItemOperationTimeout: 4h0m0s Preserve Service NodePorts: auto Uploader config: HooksAttempted: 0 HooksFailed: 0 Resource List: apps/v1/Deployment: - minimal-3csivol/minimal-3csivol(created) apps/v1/ReplicaSet: - minimal-3csivol/minimal-3csivol-7b779fbd4d(created) authorization.openshift.io/v1/RoleBinding: - minimal-3csivol/system:deployers(failed) - minimal-3csivol/system:image-builders(failed) - minimal-3csivol/system:image-pullers(failed) rbac.authorization.k8s.io/v1/RoleBinding: - minimal-3csivol/system:deployers(created) - minimal-3csivol/system:image-builders(created) - minimal-3csivol/system:image-pullers(created) snapshot.storage.k8s.io/v1/VolumeSnapshot: - minimal-3csivol/velero-volume1-t5hn4(created) - minimal-3csivol/velero-volume2-rq5s9(created) - minimal-3csivol/velero-volume3-5rb56(created) snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - csi-aws-vsc(skipped) snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-36493369-50da-476c-9f91-d068baad3531(failed) - snapcontent-824ebaba-35b0-4dd9-883d-75393e25940d(failed) - snapcontent-92af2390-9b2b-45b5-8127-d8e426aa62bb(failed) v1/ConfigMap: - minimal-3csivol/kube-root-ca.crt(failed) - minimal-3csivol/openshift-service-ca.crt(failed) v1/Namespace: - minimal-3csivol(created) v1/PersistentVolume: - pvc-685c140e-e654-4ce1-9d5c-dcfb6d5bb54a(skipped) - pvc-80e98020-fc44-4326-a80e-ed5890b4c6a8(skipped) - pvc-f5ae8bae-df65-4ffe-82c4-b0771445815f(skipped) v1/PersistentVolumeClaim: - minimal-3csivol/volume1(created) - minimal-3csivol/volume2(created) - minimal-3csivol/volume3(created) v1/Secret: - minimal-3csivol/builder-dockercfg-826rv(created) - minimal-3csivol/default-dockercfg-d22jl(created) - minimal-3csivol/deployer-dockercfg-bl4n5(created) v1/ServiceAccount: - minimal-3csivol/builder(skipped) - minimal-3csivol/default(skipped) - minimal-3csivol/deployer(skipped)
Recreates on OCP 4.15 w/ OADP-1.4
Errors: Velero: <none> Cluster: <none> Namespaces: minimal-3csivol: fail to patch dynamic PV, err: context deadline exceeded, PVC: volume1, PV: pvc-93a18e81-83a9-4f27-9e54-88e1364a56aa fail to patch dynamic PV, err: context deadline exceeded, PVC: volume2, PV: pvc-879908a2-3e03-4972-af91-7d64fff40fef fail to patch dynamic PV, err: context deadline exceeded, PVC: volume3, PV: pvc-5753872d-e2dd-4da6-a07c-23cc30f13d51
On OCP 4.15 w/ OADP-1.3.1 , Velero v.1.12.3
The backup completes, but the PVC's are never bound.
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR no-pod1 no-pod1 Completed 2024-06-05 23:35:38 +0000 UTC 2024-06-05 23:35:41 +0000 UTC 0 8 2024-06-05 23:35:38 +0000 UTC <none> no-pod2 no-pod1 Completed 2024-06-05 23:37:09 +0000 UTC 2024-06-05 23:37:12 +0000 UTC 0 8 2024-06-05 23:37:09 +0000 UTC <none>
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/oadp-operator$ remote_velero describe restore no-pod1 --details Name: no-pod1 Namespace: openshift-adp Labels: <none> Annotations: <none> Phase: Completed Total items to be restored: 33 Items restored: 33 Started: 2024-06-05 23:35:38 +0000 UTC Completed: 2024-06-05 23:35:41 +0000 UTC Warnings: Velero: <none> Cluster: could not restore, VolumeSnapshotContent "snapcontent-350013bb-5bf3-4072-82dd-7ffd27668c4a" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, VolumeSnapshotContent "snapcontent-86523191-4762-492e-a92e-e2e3e85898b1" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, VolumeSnapshotContent "snapcontent-fe61819d-e7a2-4455-bf44-7943b7f2c1f1" already exists. Warning: the in-cluster version is different than the backed-up version Namespaces: minimal-3csivol: could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, ConfigMap "openshift-service-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, RoleBinding "system:deployers" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, RoleBinding "system:image-builders" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, RoleBinding "system:image-pullers" already exists. Warning: the in-cluster version is different than the backed-up version Backup: no-pod1 Namespaces: Included: all namespaces found in the backup Excluded: <none> Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io Cluster-scoped: auto Namespace mappings: <none> Label selector: <none> Or label selector: <none> Restore PVs: auto Existing Resource Policy: <none> ItemOperationTimeout: 4h0m0s Preserve Service NodePorts: auto Resource List: apps/v1/Deployment: - minimal-3csivol/minimal-3csivol(created) apps/v1/ReplicaSet: - minimal-3csivol/minimal-3csivol-7b779fbd4d(created) authorization.openshift.io/v1/RoleBinding: - minimal-3csivol/system:deployers(failed) - minimal-3csivol/system:image-builders(failed) - minimal-3csivol/system:image-pullers(failed) rbac.authorization.k8s.io/v1/RoleBinding: - minimal-3csivol/system:deployers(created) - minimal-3csivol/system:image-builders(created) - minimal-3csivol/system:image-pullers(created) snapshot.storage.k8s.io/v1/VolumeSnapshot: - minimal-3csivol/velero-volume1-dqw68(created) - minimal-3csivol/velero-volume2-dz5j6(created) - minimal-3csivol/velero-volume3-k45p8(created) snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - csi-aws-vsc(skipped) snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-350013bb-5bf3-4072-82dd-7ffd27668c4a(failed) - snapcontent-86523191-4762-492e-a92e-e2e3e85898b1(failed) - snapcontent-fe61819d-e7a2-4455-bf44-7943b7f2c1f1(failed) v1/ConfigMap: - minimal-3csivol/kube-root-ca.crt(failed) - minimal-3csivol/openshift-service-ca.crt(failed) v1/Namespace: - minimal-3csivol(created) v1/PersistentVolume: - pvc-2853b5c5-2767-4f0f-9284-291d1a2b9329(skipped) - pvc-4bb128f8-4ced-41dc-99a5-83066597f703(skipped) - pvc-9fdb9241-86db-4e2f-bb8f-76c4ab2ee763(skipped) v1/PersistentVolumeClaim: - minimal-3csivol/volume1(created) - minimal-3csivol/volume2(created) - minimal-3csivol/volume3(created) v1/Secret: - minimal-3csivol/builder-dockercfg-jx9c7(created) - minimal-3csivol/builder-token-t6jm2(skipped) - minimal-3csivol/default-dockercfg-zcsrx(created) - minimal-3csivol/default-token-t47rk(skipped) - minimal-3csivol/deployer-dockercfg-bx7m5(created) - minimal-3csivol/deployer-token-vqc2b(skipped) v1/ServiceAccount: - minimal-3csivol/builder(skipped) - minimal-3csivol/default(skipped) - minimal-3csivol/deployer(skipped)
This is a regression in Velero behavior:
Likely Cause is https://github.com/vmware-tanzu/velero/pull/7504