Calico supports different types of overlay networks with the default on ROKS set to IPIPMode: CrossSubnet. Currently, Submariner is only verified when Calico is deployed with IPIPMode: Always (or VXLAN).

With ROKS default configuration (IPIPMode: CrossSubnet) the ingress traffic from submariner gw-node to destination worker node (where dest pod is running) isn't encapsulated in IPIP tunnel and as a result of that cross-cluster datapath is broken.

Cross-cluster datapath is broken because each ROKS cluster comes with default cluster SG, that blocks traffic sent from remote cluster (sourceIP is from remote cluster pods CIDR range).

Even If cluster SG will be updated somehow to allow inbound traffic from remote clusters, the packet will be dropped by the kernel on worker node because eth0 network interface uses strict RPF (default config). check  [1] for more details.

Bottom line we have the following options for setting Calico overlay IPIPMode :

1. Always , Pros: Submariner data-path working , Cons: datapath performance, how to persist the change
2. CrossSubnet , Pros: datapath performance , Cons: need to address datapth issue in Calico,Submariner side 

 [1] 

https://docs.google.com/document/d/1BpeU_voIVzRUotnDtOp7yKCAXrbqfgLTUG9gb_HKk5M/edit?usp=sharing