Steps taken and Log: 1) Pre-requisites: OCP AWS cluster01 (west-mesh) has been provisioned in region us-west-2 OCP AWS cluster02 (east-mesh) has been provisioned in region us-east-2 OSSM operators installed on both clusters 2) Steps [root@skondkar failover]# export MESH1_KUBECONFIG=/root/west-mesh/auth/kubeconfig [root@skondkar failover]# export MESH2_KUBECONFIG=/root/east-mesh/auth/kubeconfig [root@skondkar failover]# echo $MESH1_KUBECONFIG /root/west-mesh/auth/kubeconfig [root@skondkar failover]# echo $MESH2_KUBECONFIG /root/east-mesh/auth/kubeconfig [root@skondkar failover]# source common.sh ##### Using the following kubeconfig files: mesh1: /root/west-mesh/auth/kubeconfig mesh2: /root/east-mesh/auth/kubeconfig [root@skondkar failover]# oc1 login -u kubeadmin -p FDLaL-o23mh-CNaHp-6XikL --server=https://api.west-mesh.devcluster.openshift.com:6443 --insecure-skip-tls-verify=true Login successful. You have access to 69 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "bookinfo-ha". [root@skondkar failover]# oc2 login -u kubeadmin -p qfnua-fPvC2-LioSd-4sWhM --server=https://api.east-mesh.devcluster.openshift.com:6443 --insecure-skip-tls-verify=true Login successful. You have access to 69 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "bookinfo-ha". [root@skondkar failover]# ./setup.sh ##### Using the following kubeconfig files: mesh1: /root/west-mesh/auth/kubeconfig mesh2: /root/east-mesh/auth/kubeconfig ##### Creating projects for west-mesh Error from server (AlreadyExists): project.project.openshift.io "west-mesh-system" already exists Error from server (AlreadyExists): project.project.openshift.io "bookinfo-ha" already exists ##### Creating projects for east-mesh Error from server (AlreadyExists): project.project.openshift.io "east-mesh-system" already exists Error from server (AlreadyExists): project.project.openshift.io "bookinfo-ha" already exists ##### Installing control plane for west-mesh servicemeshcontrolplane.maistra.io/fed-export created servicemeshmemberroll.maistra.io/default created ##### Installing control plane for east-mesh servicemeshcontrolplane.maistra.io/fed-import created servicemeshmemberroll.maistra.io/default created ##### Waiting for west-mesh installation to complete servicemeshmemberroll.maistra.io/default condition met ##### Waiting for east-mesh installation to complete servicemeshmemberroll.maistra.io/default condition met [root@skondkar failover]# ------------------------ [root@skondkar failover]# ./install.sh ##### Using the following kubeconfig files: mesh1: /root/west-mesh/auth/kubeconfig mesh2: /root/east-mesh/auth/kubeconfig ##### Retrieving root certificates ##### Retrieving ingress addresses Two clusters detected; using load-balancer service for ingress WEST_MESH_ADDRESS=ad57bbfb900514a9b9a3ebe0e678d364-2521d63d99d30f8e.elb.eu-west-2.amazonaws.com WEST_MESH_DISCOVERY_PORT=8188 WEST_MESH_SERVICE_PORT=15443 EAST_MESH_ADDRESS=af5693d62e9ac40899b9b0cfca7bc117-563957cd8b33775b.elb.us-east-2.amazonaws.com EAST_MESH_DISCOVERY_PORT=8188 EAST_MESH_SERVICE_PORT=15443 ##### Enabling federation for west-mesh configmap/east-mesh-ca-root-cert created servicemeshpeer.federation.maistra.io/east-mesh created ##### Enabling federation for east-mesh configmap/west-mesh-ca-root-cert created servicemeshpeer.federation.maistra.io/west-mesh created ##### Installing bookinfo in west-mesh service/details created serviceaccount/bookinfo-details created deployment.apps/details-v1 created service/ratings created serviceaccount/bookinfo-ratings created deployment.apps/ratings-v1 created service/reviews created serviceaccount/bookinfo-reviews created deployment.apps/reviews-v1 created deployment.apps/reviews-v2 created deployment.apps/reviews-v3 created service/productpage created serviceaccount/bookinfo-productpage created deployment.apps/productpage-v1 created destinationrule.networking.istio.io/productpage created destinationrule.networking.istio.io/reviews created destinationrule.networking.istio.io/ratings created destinationrule.networking.istio.io/details created ##### Installing bookinfo in east-mesh service/details created serviceaccount/bookinfo-details created deployment.apps/details-v1 created service/ratings created serviceaccount/bookinfo-ratings created deployment.apps/ratings-v1 created service/reviews created serviceaccount/bookinfo-reviews created deployment.apps/reviews-v1 created deployment.apps/reviews-v2 created deployment.apps/reviews-v3 created service/productpage created serviceaccount/bookinfo-productpage created deployment.apps/productpage-v1 created gateway.networking.istio.io/bookinfo-gateway created virtualservice.networking.istio.io/bookinfo created destinationrule.networking.istio.io/productpage created destinationrule.networking.istio.io/reviews created destinationrule.networking.istio.io/ratings created destinationrule.networking.istio.io/details created virtualservice.networking.istio.io/reviews created ##### INSTALLATION COMPLETE { "apiVersion": "federation.maistra.io/v1", "kind": "ServiceMeshPeer", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"federation.maistra.io/v1\",\"kind\":\"ServiceMeshPeer\",\"metadata\":{\"annotations\":{},\"name\":\"east-mesh\",\"namespace\":\"west-mesh-system\"},\"spec\":{\"gateways\":{\"egress\":{\"name\":\"east-mesh-egress\"},\"ingress\":{\"name\":\"east-mesh-ingress\"}},\"remote\":{\"addresses\":[\"af5693d62e9ac40899b9b0cfca7bc117-563957cd8b33775b.elb.us-east-2.amazonaws.com\"],\"discoveryPort\":8188,\"servicePort\":15443},\"security\":{\"certificateChain\":{\"kind\":\"ConfigMap\",\"name\":\"east-mesh-ca-root-cert\"},\"clientID\":\"east-mesh.local/ns/east-mesh-system/sa/west-mesh-egress-service-account\",\"trustDomain\":\"east-mesh.local\"}}}\n" }, "creationTimestamp": "2022-07-14T18:52:32Z", "generation": 1, "name": "east-mesh", "namespace": "west-mesh-system", "resourceVersion": "158843", "uid": "cce43af5-043e-48be-9bec-40f53e87270c" }, "spec": { "gateways": { "egress": { "name": "east-mesh-egress" }, "ingress": { "name": "east-mesh-ingress" } }, "remote": { "addresses": [ "af5693d62e9ac40899b9b0cfca7bc117-563957cd8b33775b.elb.us-east-2.amazonaws.com" ], "discoveryPort": 8188, "servicePort": 15443 }, "security": { "certificateChain": { "kind": "ConfigMap", "name": "east-mesh-ca-root-cert" }, "clientID": "east-mesh.local/ns/east-mesh-system/sa/west-mesh-egress-service-account", "trustDomain": "east-mesh.local" } }, "status": { "discoveryStatus": { "active": [ { "pod": "istiod-fed-export-765cc87677-488xh", "remotes": [ { "connected": true, "lastConnected": "2022-07-14T18:52:55Z", "lastFullSync": "2022-07-14T18:52:55Z", "source": "10.129.2.12" } ], "watch": { "connected": true, "lastConnected": "2022-07-14T18:53:17Z", "lastDisconnectStatus": "503 Service Unavailable", "lastFullSync": "2022-07-14T18:53:25Z" } } ] } } } { "apiVersion": "federation.maistra.io/v1", "kind": "ServiceMeshPeer", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"federation.maistra.io/v1\",\"kind\":\"ServiceMeshPeer\",\"metadata\":{\"annotations\":{},\"name\":\"west-mesh\",\"namespace\":\"east-mesh-system\"},\"spec\":{\"gateways\":{\"egress\":{\"name\":\"west-mesh-egress\"},\"ingress\":{\"name\":\"west-mesh-ingress\"}},\"remote\":{\"addresses\":[\"ad57bbfb900514a9b9a3ebe0e678d364-2521d63d99d30f8e.elb.eu-west-2.amazonaws.com\"],\"discoveryPort\":8188,\"servicePort\":15443},\"security\":{\"certificateChain\":{\"kind\":\"ConfigMap\",\"name\":\"west-mesh-ca-root-cert\"},\"clientID\":\"west-mesh.local/ns/west-mesh-system/sa/east-mesh-egress-service-account\",\"trustDomain\":\"west-mesh.local\"}}}\n" }, "creationTimestamp": "2022-07-14T18:52:54Z", "generation": 1, "name": "west-mesh", "namespace": "east-mesh-system", "resourceVersion": "132014", "uid": "fe579253-97fb-4dc9-9a37-d264cb566b2d" }, "spec": { "gateways": { "egress": { "name": "west-mesh-egress" }, "ingress": { "name": "west-mesh-ingress" } }, "remote": { "addresses": [ "ad57bbfb900514a9b9a3ebe0e678d364-2521d63d99d30f8e.elb.eu-west-2.amazonaws.com" ], "discoveryPort": 8188, "servicePort": 15443 }, "security": { "certificateChain": { "kind": "ConfigMap", "name": "west-mesh-ca-root-cert" }, "clientID": "west-mesh.local/ns/west-mesh-system/sa/east-mesh-egress-service-account", "trustDomain": "west-mesh.local" } }, "status": { "discoveryStatus": { "active": [ { "pod": "istiod-fed-import-68d5cbd899-hn4v5", "remotes": [ { "connected": true, "lastConnected": "2022-07-14T18:53:17Z", "lastFullSync": "2022-07-14T18:53:25Z", "source": "10.131.0.25" } ], "watch": { "connected": true, "lastConnected": "2022-07-14T18:52:54Z", "lastDisconnectStatus": "Get \"http://west-mesh-egress.east-mesh-system.svc.cluster.local:8188/v1/watch\": dial tcp 172.30.215.12:8188: connect: connection refused", "lastFullSync": "2022-07-14T18:52:55Z" } } ] } } } If servicemeshpeer connection is false. Then Wait 10 minutes and then run install.sh again. OCP node failure-domain region value for cluster01 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 failure-domain.beta.kubernetes.io/region=eu-west-2 OCP node failure-domain region value for cluster02 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 failure-domain.beta.kubernetes.io/region=us-east-2 [root@skondkar failover]# ====================================================================================================== [root@skondkar failover]# oc1 apply -f export/exportedserviceset.yaml exportedserviceset.federation.maistra.io/east-mesh created [root@skondkar failover]# oc2 apply -f import/importedserviceset.yaml importedserviceset.federation.maistra.io/west-mesh created [root@skondkar failover]# oc2 -n east-mesh-system get importedservicesets west-mesh -o json { "apiVersion": "federation.maistra.io/v1", "kind": "ImportedServiceSet", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"federation.maistra.io/v1\",\"kind\":\"ImportedServiceSet\",\"metadata\":{\"annotations\":{},\"name\":\"west-mesh\",\"namespace\":\"east-mesh-system\"},\"spec\":{\"importRules\":[{\"importAsLocal\":true,\"nameSelector\":{\"alias\":{\"name\":\"ratings\",\"namespace\":\"bookinfo-ha\"},\"namespace\":\"bookinfo\"},\"type\":\"NameSelector\"}],\"locality\":{\"region\":\"us-west-2\",\"zone\":\"us-west-2a\"}}}\n" }, "creationTimestamp": "2022-07-14T19:05:17Z", "generation": 1, "name": "west-mesh", "namespace": "east-mesh-system", "resourceVersion": "137434", "uid": "44e8bfd0-5caf-4a5b-8ceb-f3102c27acf4" }, "spec": { "importRules": [ { "importAsLocal": true, "nameSelector": { "alias": { "name": "ratings", "namespace": "bookinfo-ha" }, "namespace": "bookinfo" }, "type": "NameSelector" } ], "locality": { "region": "us-west-2", "zone": "us-west-2a" } } } [root@skondkar failover]# ===================================================================================================== Wait for 10 Minutes here ====================================================================================================== [root@skondkar failover]# oc2 apply -f examples/destinationrule-failover.yaml destinationrule.networking.istio.io/failover created ================================= Verification: ++++++++++++++++++++++++++++++++++++++ Refresh cluster02 (east-mesh) boookinfo productpage. [root@skondkar failover]# curl http://bookinfo-ha-bookinfo-gateway-525eca1d5089dbdc-east-mesh-system.apps.east-mesh.devcluster.openshift.com/productpage And check cluster02 (east-mesh) bookinfo-ha ns pod ratings-v1 ratings container log for example, Server listening on: http://0.0.0.0:9080 GET /ratings/0 Check cluster01 (west-mesh) bookinfo-ha ns pod ratings-v1 ratings container log. There is no new GET request coming in cluster01 after applying the DestinationRule. ---------------------------------------------------------------------------------- Scale ratings-v1 deployment in cluster02 (east-mesh) bookinfo-ha to 0 Refresh cluster02 (east-mesh) boookinfo productpage curl http://bookinfo-ha-bookinfo-gateway-525eca1d5089dbdc-east-mesh-system.apps.east-mesh.devcluster.openshift.com/productpage Check cluster01 bookinfo-ha ns pod ratings-v1 ratings container log. Server listening on: http://0.0.0.0:9080 GET /ratings/0 Restore ratings-v1 deployment in cluster02 (east-mesh) bookinfo-ha to 1 Refresh cluster02 (east-mesh) boookinfo productpage curl http://bookinfo-ha-bookinfo-gateway-525eca1d5089dbdc-east-mesh-system.apps.east-mesh.devcluster.openshift.com/productpage Check cluster02 (east-mesh) bookinfo-ha ns pod ratings-v1 ratings container log. Server listening on: http://0.0.0.0:9080 GET /ratings/0 --------------