Description of problem:

There is a HUB cluster with RHCAM 2.12.1 that has configured two cluster for Regional DR in ODF 4.16.3 : primary ocp-prod and remote : ocp-dr

On OCP cluster "ocp-prod-mz" rbd commands execute on ceph toolbox pod hung - We found osd pods do not receive request sent to 242.0.255.X

Each osd has configured two IPs  Public address: 242.0.255.x  AND private/public-bind-addr 10.x.x.x

<redacted>

Here is a example of the debug logs of rbd command:

<redacted>

The IP 242.0.255.251 is from osd.1  but the osd logs shows iOSD.1 never got request from that IP , so the client is not reaching OSD.1 , this is issue outside of Ceph part.

<redacted> 

Also, we have run "subctl diagnose" and we get errors like

<redacted>

Version-Release number of selected component (if applicable):

RHACM 2.12.1

ODF 4.16.3

How reproducible: N/A

Steps to Reproduce:

1.  
2.  
3. ...

Actual results:

<redacted>

Additional info: