Description of problem (please be detailed as possible and provide log
snippests):

Version of all relevant components (if applicable):
ODF 4.15.0-132.stable
OCP 4.15.0-0.nightly-2024-02-13-231030
ACM 2.9.2 GA'ed
Submariner 0.16.3
ceph version 17.2.6-194.el9cp (d9f4aedda0fc0d99e7e0e06892a69523d2eb06dc) quincy (stable)

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Is there any workaround available to the best of your knowledge?

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
*Active hub at neutral site*

1. Deployed multiple rbd and cephfs backed workloads of both appset and subscription types.
2. Failedover and relocated them in such a way that they are finally running on the primary managed cluster (which is expected to host all the workloads and can go under disaster) but the apps which were failedover from C1 to C2 were relocated back to C1 and the apps which were relocated to C2 were failedover to C1 (with all nodes up and running).
Ensure that we have all workloads combinations in distinct states like deployed, failedover, relocated on C1, and a few workloads in deployed state on C2 as well.
4. Let the latest backups be taken at least 1 for all the different states of the workloads (when progression is completed and no action is going on any of the workloads). Also ensure sync for all the workloads when on active hub is working fine and cluster is healthy. Note drpc -o wide, lastGroupSyncTime, download backups from S3, etc.
5. Bring active hub completely down, move to passive hub. Restore backps, ensure velero backup reports successful restoration. Make sure both the managed clusters are successfully reported, drpolicy gets validated.
6. Wait for drpc to be restored, check if all the workloads are in their last backedup state or not.
They seem to have retained their last state which was backedup. So everything is fine so far.
7. Let IOs continue for a few hours (20-30hrs). Failover the cephfs workloads running on C2 to C1 with all nodes of C2 up and running.
8. After successful failover and cleanup, wait for sync to resume and after some time bring primary cluster down (all nodes). Bring it up after a few hours.
9. Check if drpc state is still the same and data sync for all workload is resuming as expected.
10. After a few hours, bring master nodes of primary cluster down and check drpc again.

(Older hub remains down forever and is completely unreachable).

The below CephFS app changes it's state from Relocated to Relocating without any action on it.

Before==>

busybox-workloads-5 sub-cephfs-busybox5-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Relocate Relocated Completed 2024-02-17T14:56:29Z 18h39m39.024430842s True

After==>

busybox-workloads-5 sub-cephfs-busybox5-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Relocate Relocating 2024-02-18T19:16:07Z False

Since it assumes that relocate operation for this workload is in progress, it couldn't be failedover as peer ready becomes false, while all other workloads running on primary were successfully failedover after the master nodes of the primary cluster went down.

amagrawa:hub$ drpc
NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-workloads-13 sub-rbd-busybox13-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver WaitForReadiness 2024-02-18T19:20:17Z False
busybox-workloads-14 sub-rbd-busybox14-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver WaitForReadiness 2024-02-18T19:20:25Z False
busybox-workloads-15 sub-rbd-busybox15-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver WaitForReadiness 2024-02-18T19:20:33Z False
busybox-workloads-16 sub-rbd-busybox16-placement-1-drpc 2d9h amagrawa-odf2 Deployed Completed 2024-02-16T10:12:51Z 660.371688ms True
busybox-workloads-5 sub-cephfs-busybox5-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Relocate Relocating 2024-02-18T19:16:07Z False
busybox-workloads-6 sub-cephfs-busybox6-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver Cleaning Up 2024-02-18T19:19:49Z False
busybox-workloads-7 sub-cephfs-busybox7-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver Cleaning Up 2024-02-18T19:19:59Z False
busybox-workloads-8 sub-cephfs-busybox8-placement-1-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver Cleaning Up 2024-02-18T19:20:06Z False
openshift-gitops appset-cephfs-busybox1-placement-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver Cleaning Up 2024-02-18T19:18:33Z False
openshift-gitops appset-cephfs-busybox2-placement-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver Cleaning Up 2024-02-18T19:18:38Z False
openshift-gitops appset-cephfs-busybox3-placement-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver Cleaning Up 2024-02-18T19:18:43Z False
openshift-gitops appset-cephfs-busybox4-placement-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver Cleaning Up 2024-02-18T19:18:48Z False
openshift-gitops appset-rbd-busybox10-placement-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver WaitForReadiness 2024-02-18T19:18:52Z False
openshift-gitops appset-rbd-busybox11-placement-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver WaitForReadiness 2024-02-18T19:18:58Z False
openshift-gitops appset-rbd-busybox12-placement-drpc 2d9h amagrawa-odf2 Deployed Completed 2024-02-16T10:13:47Z 571.259493ms True
openshift-gitops appset-rbd-busybox9-placement-drpc 2d9h amagrawa-prim amagrawa-odf2 Failover FailedOver WaitForReadiness 2024-02-18T19:19:17Z False

This leads to inaccessibility of this application.

Actual results: [RDR] [Hub recovery] CephFS workload changes it's state from Relocated to Relocating on node failure

Expected results: Applications should retain their original state after node failure so that they could be failedover.

Additional info: