Description of problem (please be detailed as possible and provide log
snippests):

Version of all relevant components (if applicable):
ACM 2.10.2 GA'ed
MCE 2.5.2
ODF 4.15.2-1 GA'ed
ceph version 17.2.6-209.el9cp (e9529323dd7ab3b0e8cdf84e17a1b58c2b42948c) quincy (stable)
OCP 4.15.0-0.nightly-2024-04-30-234425
Submariner 0.17.1 GA'ed
VolSync 0.9.1

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Is there any workaround available to the best of your knowledge?

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
****Active hub co-situated with primary managed cluster****

1. When we have multiple workloads (RBD and CephFS) of both subscription and appset types (pull model) in Deployed state running on primary managed cluster (C1) which goes down along with
active hub cluster during site failure at site-1, perform hub recovery and move to passive hub at site-2 (which is co-situated with secondary managed cluster C2).
2. Ensure the available managed cluster C2 is successfully imported on the RHACM console of the passive hub, and DRPolicy gets validated.
2. After DRPC is restored, recover the down managed cluster C1 and ensure it's successfully imported on the RHACM console.
4. Let IOs continue for some time (30mins-1hr) and ensure data sync is progressing well.
5. Now failover some of the workloads (with both managed clusters up and running) and relocate remaining ones to the C2 managed cluster during the eviction period timeout (which is currently set to 24hrs).

Actual results: [RDR] [Hub recovery] [Co-situated] Relocate operation and cleanup after failover remains stuck during the eviction period timeout

Hub-

oc get drpc -o wide -A
NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-workloads-101 rbd-sub-busybox101-placement-1-drpc 14h amagrawa-c2-13apr amagrawa-c1-13apr Relocate Relocating EnsuringVolumesAreSecondary 2024-05-05T17:32:04Z False
busybox-workloads-103 cephfs-sub-busybox103-placement-1-drpc 14h amagrawa-c2-13apr amagrawa-c1-13apr Relocate Relocating RunningFinalSync 2024-05-05T17:31:53Z True
openshift-gitops cephfs-appset-busybox102-placement-drpc 14h amagrawa-c2-13apr amagrawa-c1-13apr Relocate Relocating RunningFinalSync 2024-05-05T17:31:45Z True
openshift-gitops rbd-appset-busybox100-placement-drpc 14h amagrawa-c1-13apr amagrawa-c2-13apr Failover FailedOver Cleaning Up 2024-05-05T17:31:57Z False

Failover for rbd-appset-busybox100-placement-drpc worked but cleanup is stuck, however relocate of all other workloads in stuck.

failedover/relocated from C1 to C2-

C2-

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-100
NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE
volumereplication.replication.storage.openshift.io/busybox-pvc-41 14h rbd-volumereplicationclass-473128587 busybox-pvc-41 primary Primary

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/busybox-pvc-41 Bound pvc-7c5e424d-b75a-495d-8745-4d3220fc48e6 42Gi RWO ocs-storagecluster-ceph-rbd 14h Filesystem

NAME DESIREDSTATE CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/rbd-appset-busybox100-placement-drpc primary Primary

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/busybox-41-5c55b45d49-qh7v8 1/1 Running 0 13h 10.129.2.51 compute-2 <none> <none>

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-101
No resources found in busybox-workloads-101 namespace.

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-102
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/busybox-pvc-1 Bound pvc-542ab575-db38-4187-bbb0-70697ea232f3 94Gi RWX ocs-storagecluster-cephfs 3d16h Filesystem

NAME DESIREDSTATE CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/cephfs-appset-busybox102-placement-drpc secondary Secondary

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/volsync-rsync-tls-dst-busybox-pvc-1-mw285 1/1 Running 0 2m41s 10.129.2.156 compute-2 <none> <none>

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-103
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/busybox-pvc-1 Bound pvc-3256f1e5-79e3-43ff-96cb-e0b727ffcc74 94Gi RWX ocs-storagecluster-cephfs 3d16h Filesystem

NAME DESIREDSTATE CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/cephfs-sub-busybox103-placement-1-drpc secondary Secondary

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/volsync-rsync-tls-dst-busybox-pvc-1-2q7zp 1/1 Running 0 2m46s 10.129.2.155 compute-2 <none> <none>

C1-

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-100
NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE
volumereplication.replication.storage.openshift.io/busybox-pvc-41 3d16h rbd-volumereplicationclass-473128587 busybox-pvc-41 primary Primary

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/busybox-pvc-41 Bound pvc-7c5e424d-b75a-495d-8745-4d3220fc48e6 42Gi RWO ocs-storagecluster-ceph-rbd 3d16h Filesystem

NAME DESIREDSTATE CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/rbd-appset-busybox100-placement-drpc secondary Primary

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/busybox-41-5c55b45d49-fngg2 1/1 Running 2 3d16h 10.128.3.196 compute-0 <none> <none>

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-101
NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE
volumereplication.replication.storage.openshift.io/busybox-pvc-41 3d16h rbd-volumereplicationclass-473128587 busybox-pvc-41 primary Primary

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/busybox-pvc-41 Bound pvc-ea52541b-acb4-4ecb-afd6-a00925bf3583 42Gi RWO ocs-storagecluster-ceph-rbd 3d16h Filesystem

NAME DESIREDSTATE CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/rbd-sub-busybox101-placement-1-drpc secondary Primary

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/busybox-41-5c55b45d49-59tbp 1/1 Running 2 3d16h 10.128.3.198 compute-0 <none> <none>

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-102
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/busybox-pvc-1 Bound pvc-46f34593-ba2f-435c-801d-66b7371fd359 94Gi RWX ocs-storagecluster-cephfs 3d16h Filesystem

NAME DESIREDSTATE CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/cephfs-appset-busybox102-placement-drpc primary Primary

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/busybox-1-7f9b67dc95-wq4tr 1/1 Running 2 3d16h 10.128.3.208 compute-0 <none> <none>

oc get vr,pvc,vrg,pods -o wide -n busybox-workloads-103
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/busybox-pvc-1 Bound pvc-40f27ae0-e3f5-4e74-822e-0eab289f3232 94Gi RWX ocs-storagecluster-cephfs 3d16h Filesystem

NAME DESIREDSTATE CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/cephfs-sub-busybox103-placement-1-drpc primary Primary

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/busybox-1-7f9b67dc95-v6h4s 1/1 Running 2 3d16h 10.128.3.209 compute-0 <none> <none>

Logs- http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-aman/05may24/

Expected results: Admin should be able to successfully failover/relocate the workloads independent of eviction period timeout post hub recovery.

Additional info: