csi-cephfsplugin-provisioner log -------------------------------- E0517 15:32:59.011508 1 utils.go:210] ID: 276025 Req-ID: pvc-64ecf89c-f937-4f68-bbae-2854e085e261 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress E0517 15:33:04.996695 1 utils.go:210] ID: 276027 Req-ID: pvc-9b1a47bb-4830-4dee-b7ba-fb515d6a8cd3 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress E0517 15:33:38.136337 1 controllerserver.go:238] ID: 276050 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a CreateVolumeRequest validation failed: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:38.136346 1 utils.go:210] ID: 276050 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a GRPC error: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:38.639018 1 controllerserver.go:238] ID: 276052 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a CreateVolumeRequest validation failed: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:38.639029 1 utils.go:210] ID: 276052 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a GRPC error: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:39.127350 1 utils.go:210] ID: 276053 Req-ID: pvc-fd12495e-85fb-480d-93a3-e6d9e1ba9da4 GRPC error: rpc error: code = Aborted desc = clone from snapshot is pending E0517 15:33:39.641906 1 controllerserver.go:238] ID: 276055 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a CreateVolumeRequest validation failed: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:39.641916 1 utils.go:210] ID: 276055 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a GRPC error: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:39.654421 1 utils.go:210] ID: 276054 Req-ID: pvc-fd12495e-85fb-480d-93a3-e6d9e1ba9da4 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress E0517 15:33:40.680794 1 utils.go:210] ID: 276056 Req-ID: pvc-fd12495e-85fb-480d-93a3-e6d9e1ba9da4 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress E0517 15:33:41.644234 1 controllerserver.go:238] ID: 276057 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a CreateVolumeRequest validation failed: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:41.644239 1 utils.go:210] ID: 276057 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a GRPC error: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:42.707342 1 utils.go:210] ID: 276058 Req-ID: pvc-fd12495e-85fb-480d-93a3-e6d9e1ba9da4 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress E0517 15:33:45.646754 1 controllerserver.go:238] ID: 276059 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a CreateVolumeRequest validation failed: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:45.646762 1 utils.go:210] ID: 276059 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a GRPC error: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:46.733917 1 utils.go:210] ID: 276060 Req-ID: pvc-fd12495e-85fb-480d-93a3-e6d9e1ba9da4 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress E0517 15:33:53.650216 1 controllerserver.go:238] ID: 276062 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a CreateVolumeRequest validation failed: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:53.650224 1 utils.go:210] ID: 276062 Req-ID: pvc-06613b37-fe6b-48f6-86ac-d56b9e1f705a GRPC error: rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source E0517 15:33:54.759430 1 utils.go:210] ID: 276063 Req-ID: pvc-fd12495e-85fb-480d-93a3-e6d9e1ba9da4 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress E0517 15:34:01.993019 1 utils.go:210] ID: 276064 Req-ID: pvc-3d4b4707-c605-4dbd-8d3a-b4dfc3bc9188 GRPC error: rpc error: code = Aborted desc = clone from snapshot is pending Velero pod log -------------- ime="2023-05-17T15:33:28Z" level=error msg="Error updating download request" controller=download-request downloadRequest=openshift-adp/dm-backup-fs-10pods-multisnaps-2-6e834056-2d64-4191-924a-3f2af741e004 error="downloadrequests.velero.io \"dm-backup-fs-10pods-multisnaps-2-6e834056-2d64-4191-924a-3f2af741e004\" not found" logSource="/remote-source/velero/app/pkg/controller/download_request_controller.go:100" time="2023-05-17T15:33:29Z" level=error msg="Error updating download request" controller=download-request downloadRequest=openshift-adp/dm-backup-fs-10pods-multisnaps-2-59a817d1-ae34-4b73-ba7c-caaaf520a47c error="downloadrequests.velero.io \"dm-backup-fs-10pods-multisnaps-2-59a817d1-ae34-4b73-ba7c-caaaf520a47c\" not found" logSource="/remote-source/velero/app/pkg/controller/download_request_controller.go:100" time="2023-05-17T15:33:29Z" level=error msg="Error updating download request" controller=download-request downloadRequest=openshift-adp/dm-backup-fs-10pods-multisnaps-2-20f78eb5-a297-4ce3-963b-e00c852d20c4 error="downloadrequests.velero.io \"dm-backup-fs-10pods-multisnaps-2-20f78eb5-a297-4ce3-963b-e00c852d20c4\" not found" logSource="/remote-source/velero/app/pkg/controller/download_request_controller.go:100" vsm pod log ----------- [root@f01-h07-000-r640 20230517_153143]# grep -i error vsm.log | grep vsb-rcxtq 2023-05-17T15:33:18Z INFO Error from batch reconcile: Operation cannot be fulfilled on volumesnapshotbackups.datamover.oadp.openshift.io "vsb-rcxtq": the object has been modified; please apply your changes to the latest version and try again {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup": {"name":"vsb-rcxtq","namespace":"perf-busy-data-cephfs-10pods"}, "namespace": "perf-busy-data-cephfs-10pods", "name": "vsb-rcxtq", "reconcileID": "5b59185f-7197-4ad4-a053-5dc7d31d492f", "vsb": "perf-busy-data-cephfs-10pods/vsb-rcxtq"} 2023-05-17T16:34:01Z INFO Error from batch reconcile: cannot obtain source volumesnapshot perf-busy-data-cephfs-10pods/velero-pvc-busy-data-fs-10pods-7-xj5qq {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup": {"name":"vsb-rcxtq","namespace":"perf-busy-data-cephfs-10pods"}, "namespace": "perf-busy-data-cephfs-10pods", "name": "vsb-rcxtq", "reconcileID": "f41b7c2f-0369-4bcd-ad35-9afc25ecfd07", "vsb": "perf-busy-data-cephfs-10pods/vsb-rcxtq"} 2023-05-17T16:34:01Z ERROR Reconciler error {"controller": "volumesnapshotbackup", "controllerGroup": "datamover.oadp.openshift.io", "controllerKind": "VolumeSnapshotBackup", "VolumeSnapshotBackup": {"name":"vsb-rcxtq","namespace":"perf-busy-data-cephfs-10pods"}, "namespace": "perf-busy-data-cephfs-10pods", "name": "vsb-rcxtq", "reconcileID": "f41b7c2f-0369-4bcd-ad35-9afc25ecfd07", "error": "backup failed. Marking volumeSnapshotBackup as partiallyFailed"} CR backup describe ------------------ Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-a3bece79-b51d-4167-8400-4ad8650459f6: Backup Item Action Plugin: velero.io/vsm-volumesnapshotcontent-backupper Operation ID: perf-busy-data-cephfs-10pods/vsb-rcxtq Items to Update: volumesnapshotbackups.datamover.oadp.openshift.io perf-busy-data-cephfs-10pods/vsb-rcxtq Phase: Failed Operation Error: Asynchronous action timed out Progress description: Phase: InProgress BatchingStatus: Processing Created: 2023-05-17 15:33:18 +0000 UTC Started: 2023-05-17 15:33:18 +0000 UTC Updated: 2023-05-17 16:33:27 +0000 UTC Backups status -------------- NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR dm-backup-fs-10pods-multisnaps-1 Completed 0 0 2023-05-17 14:56:36 +0000 UTC 29d example-velero-1 dm-backup-fs-10pods-multisnaps-2 Completed 0 0 2023-05-17 15:13:39 +0000 UTC 29d example-velero-1 dm-backup-fs-10pods-multisnaps-3 PartiallyFailed 1 0 2023-05-17 15:31:44 +0000 UTC 29d example-velero-1 [root@f01-h07-000-r640 playbooks]# echo "PV" ; oc get pv -A | grep -v "openshift-storage" ; echo "PVC" ; oc get pvc -A | grep -v "openshift-storage" ; echo "OADP POODS" ; oc get pods ; echo "VSB NS" ; oc get vsb -nperf-busy-data-cephfs-10pods PV NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-0629e05d-4f27-4608-b485-4d8c4ad064c0 100Gi RWX Delete Bound openshift-image-registry/image-registry-storage ocs-storagecluster-cephfs 42d pvc-1a8a982d-fbc4-498c-ba45-b5aa7654ec65 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-1 ocs-storagecluster-cephfs 78m pvc-226b6fb1-165f-4358-be9f-192c29c6c497 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-2 ocs-storagecluster-cephfs 78m pvc-2837db00-e7ab-4680-8b70-60915065b6a0 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-10 ocs-storagecluster-cephfs 76m pvc-28dd22ad-4f3d-41d8-9eb0-d1b1f27f9534 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-9 ocs-storagecluster-cephfs 77m pvc-4901b19f-58a2-4cd5-86bb-575b1af0d7d3 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-5 ocs-storagecluster-cephfs 77m pvc-956e134d-5156-4710-a69b-f354956d85dd 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-3 ocs-storagecluster-cephfs 77m pvc-a18b52bc-b752-4538-aa03-b9c7a2d20c55 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-8 ocs-storagecluster-cephfs 77m pvc-da764e81-58ee-41b4-b28d-6ba0b36f48ba 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-7 ocs-storagecluster-cephfs 77m pvc-e3913715-94c4-4346-94aa-94528dbee9c9 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-4 ocs-storagecluster-cephfs 77m pvc-f9cd4159-a393-4492-b0d8-cdad1e8875d3 6Gi ROX Delete Bound openshift-adp/snapcontent-a3bece79-b51d-4167-8400-4ad8650459f6-pvc ocs-storagecluster-cephfs-shallow 34m pvc-ff66aa85-992b-492f-a4bf-d9d20f0bef2b 6Gi RWO Delete Bound perf-busy-data-cephfs-10pods/pvc-busy-data-fs-10pods-6 ocs-storagecluster-cephfs 77m PVC NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE openshift-adp snapcontent-a3bece79-b51d-4167-8400-4ad8650459f6-pvc Bound pvc-f9cd4159-a393-4492-b0d8-cdad1e8875d3 6Gi ROX ocs-storagecluster-cephfs-shallow 34m openshift-adp volsync-vsb-rcxtq-rep-src-cache Pending ocs-storagecluster-cephfs 34m openshift-image-registry image-registry-storage Bound pvc-0629e05d-4f27-4608-b485-4d8c4ad064c0 100Gi RWX ocs-storagecluster-cephfs 42d perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-1 Bound pvc-1a8a982d-fbc4-498c-ba45-b5aa7654ec65 6Gi RWO ocs-storagecluster-cephfs 78m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-10 Bound pvc-2837db00-e7ab-4680-8b70-60915065b6a0 6Gi RWO ocs-storagecluster-cephfs 76m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-2 Bound pvc-226b6fb1-165f-4358-be9f-192c29c6c497 6Gi RWO ocs-storagecluster-cephfs 78m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-3 Bound pvc-956e134d-5156-4710-a69b-f354956d85dd 6Gi RWO ocs-storagecluster-cephfs 77m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-4 Bound pvc-e3913715-94c4-4346-94aa-94528dbee9c9 6Gi RWO ocs-storagecluster-cephfs 77m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-5 Bound pvc-4901b19f-58a2-4cd5-86bb-575b1af0d7d3 6Gi RWO ocs-storagecluster-cephfs 77m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-6 Bound pvc-ff66aa85-992b-492f-a4bf-d9d20f0bef2b 6Gi RWO ocs-storagecluster-cephfs 77m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-7 Bound pvc-da764e81-58ee-41b4-b28d-6ba0b36f48ba 6Gi RWO ocs-storagecluster-cephfs 77m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-8 Bound pvc-a18b52bc-b752-4538-aa03-b9c7a2d20c55 6Gi RWO ocs-storagecluster-cephfs 77m perf-busy-data-cephfs-10pods pvc-busy-data-fs-10pods-9 Bound pvc-28dd22ad-4f3d-41d8-9eb0-d1b1f27f9534 6Gi RWO ocs-storagecluster-cephfs 77m OADP POODS NAME READY STATUS RESTARTS AGE openshift-adp-controller-manager-bbb5b6c68-8r24t 1/1 Running 0 5d20h velero-7948845f7-xq2sd 1/1 Running 0 86m volsync-src-vsb-rcxtq-rep-src-g64nr 0/1 Pending 0 34m volume-snapshot-mover-54f9486f7-qrh7f 1/1 Running 0 9h vsb-rcxtq-pod 1/1 Running 0 34m VSB NS NAME PVC NAME VOLUMESNAPSHOTCONTENT PHASE BATCHINGSTATUS AGE vsb-5v789 pvc-busy-data-fs-10pods-5 snapcontent-787c0974-a516-4941-937d-a891ab0ab070 Completed Completed 35m vsb-c4w9c pvc-busy-data-fs-10pods-9 snapcontent-3297354c-804c-42cc-96f7-35c44598ce1c Completed Completed 34m vsb-glbxh pvc-busy-data-fs-10pods-8 snapcontent-c84e481d-dd05-455c-a092-27866d6fdaed Completed Completed 34m vsb-j4l7n pvc-busy-data-fs-10pods-4 snapcontent-cca79277-08f0-421e-8bcb-991844c0532e Completed Completed 35m vsb-mhh79 pvc-busy-data-fs-10pods-3 snapcontent-a4ecfbcb-2812-4677-a8f3-a1dfb903a898 Completed Completed 35m vsb-rcxtq pvc-busy-data-fs-10pods-7 snapcontent-a3bece79-b51d-4167-8400-4ad8650459f6 InProgress Processing 35m vsb-s52mh pvc-busy-data-fs-10pods-10 snapcontent-7051e809-4f1c-4627-9d6e-9b96e6594b02 Completed Completed 36m vsb-vcsrm pvc-busy-data-fs-10pods-6 snapcontent-4b64ee13-5ada-4090-9de4-bab02b76074c Completed Completed 35m vsb-vl4bj pvc-busy-data-fs-10pods-2 snapcontent-4335cd8c-941f-4400-9445-04355b71733e Completed Completed 35m vsb-zgkj9 pvc-busy-data-fs-10pods-1 snapcontent-663c91b5-274e-4e79-918c-079d5d55b59b Completed Completed 36m EVENTS ------ [root@f01-h07-000-r640 playbooks]# oc get events -nopenshift-adp --sort-by=.metadata.creationTimestamp | grep pvc-f9cd4159-a393-4492-b0d8-cdad1e8875d3 35m Normal ProvisioningSucceeded persistentvolumeclaim/snapcontent-a3bece79-b51d-4167-8400-4ad8650459f6-pvc Successfully provisioned volume pvc-f9cd4159-a393-4492-b0d8-cdad1e8875d3 34m Normal SuccessfulAttachVolume pod/vsb-rcxtq-pod AttachVolume.Attach succeeded for volume "pvc-f9cd4159-a393-4492-b0d8-cdad1e8875d3" 34m Warning FileSystemResizeFailed pod/vsb-rcxtq-pod MountVolume.NodeExpandVolume failed for volume "pvc-f9cd4159-a393-4492-b0d8-cdad1e8875d3" requested read-only file system 34m Warning FileSystemResizeFailed persistentvolumeclaim/snapcontent-a3bece79-b51d-4167-8400-4ad8650459f6-pvc MountVolume.NodeExpandVolume failed for volume "pvc-f9cd4159-a393-4492-b0d8-cdad1e8875d3" requested read-only file system