Description of problem (please be detailed as possible and provide log
snippests):
In ODF 4.16 cluster, configured to use custom snapshotter image, encountered an issue while attempting to create VolumeGroupSnapshot.
Stale volumesnapshot were observed when creating second volumegroupsnapshot.

Version of all relevant components (if applicable):
OCP: 4.16.0-0.nightly-2024-04-08-024331
ODF: 4.16.0-73.stable
ROOK_CSI_SNAPSHOTTER_IMAGE: "madhupr001/csi-snapshotter:v1"

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:

1. Enable feature-gate and other temporary pre-req to use custom snapshotter image
2. Create volumegroupsnapshotclass
3. Create PVC (1Gi) with desired label (In my case "group: testing")
4. Create volumegroupsnapshot with label selector and check volumesnapshot output
$ oc get volumesnapshot -A
NAMESPACE NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
openshift-storage snapshot-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26 true snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26 1Gi snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26 19m 19m

5. Create another PVC (3Gi) with same label used above
6. Create another volumegroupsnapshot with same label selector and check volumesnapshot output
$ oc get volumesnapshot -A
NAMESPACE NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
openshift-storage snapshot-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.54 true snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.54 snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.54 26s 26s
openshift-storage snapshot-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.55 true snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.55 1Gi snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.55 25s 25s
openshift-storage snapshot-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26 true snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26 1Gi snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26 31m 31m
openshift-storage snapshot-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.53 true snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.53 3Gi snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.53 27s 27s
openshift-storage snapshot-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.55 true snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.55 3Gi snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.55 25s 25s

Actual results:
Stale resources in volumesnapshot output

Expected results:
No stale resources in volumesnapshot output

Additional info: