Description of problem:

Version-Release number of selected component (if applicable):

OCP 4.14.0-0.nightly-2023-10-18-004928
advanced-cluster-management.v2.9.0-188
ODF 4.14.0-156
ceph version 17.2.6-148.el9cp (badc1d27cb07762bea48f6554ad4f92b9d3fbb6b) quincy (stable)
Submariner image: brew.registry.redhat.io/rh-osbs/iib:599799
ACM 2.9.0-DOWNSTREAM-2023-10-18-17-59-25

How reproducible:

Steps to Reproduce:

1. Create a Regional DR setup with cephfs based workloads and try the steps mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2246834.

Actual results: Data sync for cephfs workloads is affected as the dst pods remain stuck.

Expected results: dst pods for cephfs workloads should remain stuck and data sync should continue as expected b/w the managed cluster connected via submariner on a Regional DR setup.

Additional info:

When bug https://bugzilla.redhat.com/show_bug.cgi?id=2246834 was raised, the workaround was to reboot the nodes which worked fine but then the dst pods for cephfs backed workloads were stuck for long hours. This was due to the submariner issue.
Thread- https://redhat-internal.slack.com/archives/C0134E73VH6/p1698745373761259?thread_ts=1698669311.362309&cid=C0134E73VH6

GChat thread- https://chat.google.com/room/AAAAqWkMm2s/5SOd8QUuafM