Description of problem (please be detailed as possible and provide log
snippests): Cephfs and rbd workloads running on C1 were relocated to C2.
After relocation completed, it was found that older volumesnapshots from C2 (new primary) still exists and aren't getting deleted after relocate completes.

Version of all relevant components (if applicable):
OCP 4.13.0-0.nightly-2023-06-20-224158
Submariner v0.15.0-rc1
ACM 2.8 Ga'ed
ODF 4.13.0-219.snaptrim

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Is there any workaround available to the best of your knowledge?

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. On a RDR setup, deploy cephfs workloads and run it for some time (a few days in this case).
2. Check volume snapshots before Relocate on both primary & secondary.
3. Relocate the cephfs based workload to secondary cluster
4. Check volume snapshots after Relocate on both primary & secondary, here new volume snapshots will get created on new primary and new secondary but old snapshots will also be present on new primary.

Actual results:

C2 (new primary)-

amagrawa:~$ oc get volumesnapshots -n busybox-1
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
volsync-busybox-pvc-1-dst-20230628114930 true volsync-busybox-pvc-1-dst 94Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-e5bf4f46-862f-4b22-b735-179897e8e3f3 91m 91m
volsync-busybox-pvc-1-src true busybox-pvc-1 94Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-b6be69e2-9977-440f-804f-145fa3617f18 54s 58s
volsync-busybox-pvc-10-dst-20230628114929 true volsync-busybox-pvc-10-dst 87Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-4b11e1e7-9a38-41a6-b8b7-87a43e405571 91m 91m
volsync-busybox-pvc-10-src true busybox-pvc-10 87Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-6a9770ee-a7b2-4f81-9227-d187d63e4aa6 52s 58s
volsync-busybox-pvc-11-dst-20230628114929 true volsync-busybox-pvc-11-dst 33Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-ff2ec80c-aa20-4976-ad9e-bcdfbf1f9a53 91m 91m
volsync-busybox-pvc-11-src true busybox-pvc-11 33Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-d5cac59e-4dee-4806-ab37-fcad087a525b 44s 58s
volsync-busybox-pvc-12-dst-20230628114930 true volsync-busybox-pvc-12-dst 147Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-65e4a75b-8155-4443-b272-03796d5a2cc2 91m 91m
volsync-busybox-pvc-13-dst-20230628114924 true volsync-busybox-pvc-13-dst 77Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-e9da5c28-5d7a-40db-8fd7-b89590c1683a 91m 91m
volsync-busybox-pvc-13-src true busybox-pvc-13 77Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-f3b2f06d-6332-4627-9d4d-b665d933d4c9 46s 58s
volsync-busybox-pvc-14-dst-20230628114943 true volsync-busybox-pvc-14-dst 70Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-321ddcd7-8f50-47d1-810e-03a07492b64f 91m 91m
volsync-busybox-pvc-14-src true busybox-pvc-14 70Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-40dcc0c9-cbe3-43b6-b661-3251fc0af945 48s 58s
volsync-busybox-pvc-15-dst-20230628114928 true volsync-busybox-pvc-15-dst 131Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-528f57e4-8368-41d0-8e13-eded568dd5c0 91m 91m
volsync-busybox-pvc-16-dst-20230628114936 true volsync-busybox-pvc-16-dst 127Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-c8d6b459-b5bb-4954-8534-9084e10b79d4 91m 91m
volsync-busybox-pvc-16-src true busybox-pvc-16 127Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-fb9db145-6474-4195-8e0f-e1cc4fc0a4b0 48s 58s
volsync-busybox-pvc-17-dst-20230628114928 true volsync-busybox-pvc-17-dst 58Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-e1ad9f1c-f102-4b96-a855-1f4118f0a08a 91m 91m
volsync-busybox-pvc-17-src true busybox-pvc-17 58Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-3003b171-447e-4acc-bcb0-edb795d8162d 56s 58s
volsync-busybox-pvc-18-dst-20230628114928 true volsync-busybox-pvc-18-dst 123Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-b1a6d131-86bc-4e42-9f1c-70d5dbb4788f 91m 91m
volsync-busybox-pvc-18-src true busybox-pvc-18 123Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-0b5acf71-ff80-4b01-8ace-73743abef61d 57s 58s
volsync-busybox-pvc-19-dst-20230628114935 true volsync-busybox-pvc-19-dst 61Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-bb808195-741f-48b0-9b3d-322c6d240848 91m 91m
volsync-busybox-pvc-19-src true busybox-pvc-19 61Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-4261109d-4fdb-4513-8301-e77b63dca46e 53s 58s
volsync-busybox-pvc-2-dst-20230628114937 true volsync-busybox-pvc-2-dst 44Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-c66fb84a-692d-4815-8dde-47c4494f8efb 91m 91m
volsync-busybox-pvc-20-dst-20230628114929 true volsync-busybox-pvc-20-dst 33Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-95207153-c398-4169-a90d-4afdc337aafa 91m 91m
volsync-busybox-pvc-3-dst-20230628114925 true volsync-busybox-pvc-3-dst 76Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-bcca84f6-44af-4dc6-9168-6123d0fdb9bf 91m 91m
volsync-busybox-pvc-3-src true busybox-pvc-3 76Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-4c31e0a1-0649-4355-a95b-a921b0bcfdf0 44s 58s
volsync-busybox-pvc-4-dst-20230628114925 true volsync-busybox-pvc-4-dst 144Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-d5cc382b-01fa-4099-9855-5201ce5af05a 91m 91m
volsync-busybox-pvc-4-src true busybox-pvc-4 144Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-be0c6c20-ab29-4932-81e2-789138ad1efa 43s 58s
volsync-busybox-pvc-5-dst-20230628114941 true volsync-busybox-pvc-5-dst 107Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-3495ead9-a978-46dd-b548-2ab4699cfd91 91m 91m
volsync-busybox-pvc-6-dst-20230628114943 true volsync-busybox-pvc-6-dst 123Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-754d6de0-b975-45ca-a016-4e942ed7cea0 91m 91m
volsync-busybox-pvc-6-src true busybox-pvc-6 123Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-c1ff4c09-c660-40a2-b59f-c04413bb5abb 49s 58s
volsync-busybox-pvc-7-dst-20230628114906 true volsync-busybox-pvc-7-dst 90Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-ea468ba0-1f76-4b61-af3b-ef1cbeec78eb 91m 91m
volsync-busybox-pvc-7-src true busybox-pvc-7 90Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-fa995983-8123-410f-bb5f-d103f44e4559 45s 58s
volsync-busybox-pvc-8-dst-20230628114925 true volsync-busybox-pvc-8-dst 91Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-99c1a48a-67c2-4a50-b918-88dfd2ba09a7 91m 91m
volsync-busybox-pvc-9-dst-20230628114917 true volsync-busybox-pvc-9-dst 111Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-994e9732-14ec-4c83-a6ed-b5e30f55ed3d 91m 91m
volsync-busybox-pvc-9-src true busybox-pvc-9 111Gi ocs-storagecluster-cephfsplugin-snapclass snapcontent-eb9ca83b-a1f7-4054-a6ef-c5cf15c8320d 46s 58s

C1 (new secondary)-

amagrawa:~$ oc get volumesnapshotcontents -n busybox-1
NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE
snapcontent-0068eddc-3df0-4363-8c82-f64f3eb9ca72 true 81604378624 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-3-dst-20230628131044 busybox-1 5m15s
snapcontent-04d0fa41-f81e-4b14-9fac-5c1c0aeaed07 true 140660178944 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-15-dst-20230628131102 busybox-1 4m47s
snapcontent-09b10b26-a7b1-499f-b12e-1f72d9faa93d true 114890375168 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-5-dst-20230628131117 busybox-1 4m29s
snapcontent-0bc26a82-0932-4137-9e9d-49135d22bcd6 true 47244640256 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-2-dst-20230628131055 busybox-1 5m2s
snapcontent-1e2236fd-81a0-4d38-9d62-8d8b59f13438 true 82678120448 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-13-dst-20230628131109 busybox-1 4m38s
snapcontent-247936dd-bafb-4fb3-b025-bccaf073bb23 true 119185342464 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-9-dst-20230628131102 busybox-1 4m48s
snapcontent-47ff8272-4422-46d2-880b-ace78f2ac1a1 true 154618822656 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-4-dst-20230628131101 busybox-1 4m54s
snapcontent-4e02dca2-dac3-4029-bff5-487bc5778843 true 35433480192 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-11-dst-20230628131055 busybox-1 5m2s
snapcontent-59fc07d8-4e1e-4318-8dca-bb19e9c3ff5d true 100931731456 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-1-dst-20230628131101 busybox-1 4m54s
snapcontent-8278dd6f-5656-4c5e-bf2d-2815e4439c8d true 132070244352 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-18-dst-20230628131107 busybox-1 4m39s
snapcontent-88d3fe61-b139-4d20-b308-ae7c05477bd0 true 62277025792 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-17-dst-20230628131102 busybox-1 4m48s
snapcontent-8e029ff5-2c3f-4b56-8a3a-b5dcb6a63995 true 93415538688 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-10-dst-20230628131055 busybox-1 5m3s
snapcontent-a329842e-89d3-49df-b10e-a3889ce6ecf7 true 136365211648 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-16-dst-20230628131057 busybox-1 4m58s
snapcontent-bcbf7b54-912b-4e7f-b4cc-9436e15bec4b true 157840048128 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-12-dst-20230628131055 busybox-1 5m2s
snapcontent-dd5d10e6-2a45-48d4-a7cd-fecc03efaff4 true 35433480192 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-20-dst-20230628131108 busybox-1 4m39s
snapcontent-e7d9bbc4-e6c6-4d54-81b9-dd009b22877d true 97710505984 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-8-dst-20230628131112 busybox-1 4m34s
snapcontent-eb8f9dab-6d77-4e20-9643-8a467fe8b9c8 true 75161927680 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-14-dst-20230628131056 busybox-1 5m
snapcontent-ed62dcad-1703-4178-b0e7-70ccd79f7427 true 132070244352 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-6-dst-20230628131101 busybox-1 4m52s
snapcontent-eff8dac2-d955-4d02-a64d-6fc087fa7a3f true 96636764160 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-7-dst-20230628131056 busybox-1 5m1s
snapcontent-fc2364cf-218c-406c-afbe-dd59f4b9e312 true 65498251264 Delete openshift-storage.cephfs.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass volsync-busybox-pvc-19-dst-20230628131054 busybox-1 5m5s

Expected results: Older volume snapshots from new primary should be removed.

Additional info: