Description of problem:

The connection status of the submariner is degraded for Provider RDR setup. While installing Submarinar 'Globalnet' checkbox was tick marked.

The setup details:
We have used 2 Provider clusters: ibm-baremetal1 and ibm-baremetal3
with 2 hcp clusters on each---
ibm-baremetal1-hcp418-bm1-a and ibm-baremetal1-hcp418-bm1-b hcp clusters on ibm-baremetal1

ibm-baremetal3-hcp418-bm3-a and ibm-baremetal3-hcp418-bm3-b hcp clusters on ibm-baremetal3

Version-Release number of selected component (if applicable):

OCP: 4.18.0-ec.3
OCS: 4.18.0-83

How reproducible:

It is reproducible

Steps to Reproduce:

1. Create 2 Provider clusters with 2 hcp clusters on each
2. Create a ACM hub
3. Import the provider and hosted clusters to ACM hub
4. Install Submarinar add-on for all the clusters with globalnet ticked

Actual results:

The connection status of the submariner is degraded for Provider RDR setup

Expected results:

The connection status of the submariner should be healthy.

Additional info:

We have received some suggestions to try on have tried:
Approach 1:
-----------
1) The hosting cluster (e.g baremetal1/3) is already setup with required MetalLB, IPAddressPool, update to ingress controller (at least on bm-1 which I investigated), as per the steps outlined here: *  https://ibmdocs-test.dcs.ibm.com/docs/en/sfhs/2.9.x?topic=plane-prerequisites-hosted-control-clusters

• (additional ref in case)

2) Address similar gaps in the hosted clusters, i.e creation of required MetalLB, IPAddressPool, ... to complete the setup, see * https://ibmdocs-test.dcs.ibm.com/docs/en/sfhs/2.9.x?topic=clusters-deploying-bare-metal-fusion-data-foundation (step 11)

• (additional ref in case)
• NOTE: we would need 2 metallb IPs here, one for ingress and one for the submariner LB in step (3)

3) Additionally we need to shift the submariner gateway to a LoadBalancer service, this can be done via the CLI using this reference4) Test submariner connectivity between and it does not work between hosted clusters on different hosting clusters.

• bm1-a to bm3-a

But could not succeed, connection is not established.
 
RH slack thread - https://redhat-internal.slack.com/archives/C0134E73VH6/p1733237680559359?thread_ts=1732190525.902269&cid=C0134E73VH6

IBM slack thread - +https://ibm-systems-storage.slack.com/archives/C07QV45B6N4/p1733324013916009+