Executing tests against SMCP version 2.3 Found TestMultiClusterFederationFailover in file ./pkg/tests/ossm-federation/federation_failover_test.go Resetting cluster by deleting namespaces used in the test suite Output dir: /Users/frherrer/repos/maistra-test-tool/tests/result-20230427165313/v2.3 ====== Executing TestMultiClusterFederationFailover against SMCP v2.3 === RUN TestMultiClusterFederationFailover federation_common.go:63: federation_common.go:63: STEP 1: Create projects for west-mesh and east-mesh federation_common.go:64: Creating namespaces: [west-mesh-system bookinfo-ha] federation_common.go:65: Creating namespaces: [east-mesh-system bookinfo-ha] federation_common.go:67: federation_common.go:67: STEP 2: Install control planes for west-mesh and east-mesh federation_common.go:159: Install ServiceMeshControlPlane west-mesh in namespace west-mesh-system federation_common.go:164: Create ServiceMeshMemberRoll federation_common.go:159: Install ServiceMeshControlPlane east-mesh in namespace east-mesh-system federation_common.go:164: Create ServiceMeshMemberRoll federation_common.go:70: federation_common.go:70: STEP 3: Wait for west-mesh and east-mesh installation to complete federation_common.go:71: Wait for condition Ready on smcp west-mesh-system/west-mesh... federation_common.go:71: FAILURE: Command failed: oc wait -n west-mesh-system smcp/west-mesh --for condition=Ready --timeout 10s error: timed out waiting for the condition on servicemeshcontrolplanes/west-mesh error: exit status 1 federation_common.go:71: Wait for condition Ready on smcp west-mesh-system/west-mesh... federation_common.go:71: SUCCESS: Condition Ready met by smcp west-mesh-system/west-mesh federation_common.go:72: Wait for condition Ready on smcp east-mesh-system/east-mesh... federation_common.go:72: SUCCESS: Condition Ready met by smcp east-mesh-system/east-mesh federation_common.go:74: federation_common.go:74: STEP 4: Retrieve peer addresses and ports federation_common.go:81: Using LoadBalancer service for ingress federation_common.go:88: west-mesh: address: af2a1ec0ef14141539acc46bb9a40ec2-9fb1a34f7c9efea9.elb.us-east-2.amazonaws.com; discovery port: 8188, service port: 15443 federation_common.go:89: east-mesh: address: a1b1851f97f3f4a45b35932a4369b2ab-cb7325016d276d0a.elb.eu-west-1.amazonaws.com; discovery port: 8188, service port: 15443 federation_common.go:91: federation_common.go:91: STEP 5: Retrieve root certificates federation_common.go:153: Get key root-cert.pem from ConfigMap istio-ca-root-cert federation_common.go:153: Get key root-cert.pem from ConfigMap istio-ca-root-cert federation_common.go:95: federation_common.go:95: STEP 6: Install ServiceMeshPeer and ExportedServiceSet in west-mesh federation_common.go:100: federation_common.go:100: STEP 7: Install ServiceMeshPeer and ImportedServiceSet in east-mesh federation_failover_test.go:76: federation_failover_test.go:76: STEP 8: Install bookinfo in west-mesh federation_failover_test.go:80: federation_failover_test.go:80: STEP 9: Install bookinfo in east-mesh federation_failover_test.go:86: federation_failover_test.go:86: STEP 10: Install fail-over DestinationRule for ratings service in east-mesh federation_common.go:107: federation_common.go:107: STEP 11: Wait for all bookinfo pods in west-mesh and east-mesh to be ready federation_common.go:111: federation_common.go:111: STEP 12: Check if west-mesh and east-mesh are connected to each other federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 1/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 2/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 3/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 4/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 5/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 6/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 7/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 8/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 9/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 10/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 11/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 12/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 13/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 14/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 15/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 16/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 17/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 18/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 19/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 20/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 21/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 22/60 failed. Retrying in 10s... federation_common.go:113: FAILURE: west-mesh is not connected to east-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:118: FAILURE: east-mesh is not connected to west-mesh; expected to find the string '"connected": true' in the output, but it wasn't found federation_common.go:112: --- Attempt 23/60 failed. Retrying in 10s... federation_common.go:113: SUCCESS: west-mesh is connected to east-mesh federation_common.go:118: SUCCESS: east-mesh is connected to west-mesh federation_common.go:112: --- Attempt 24/60 successful; total time: 247.63s federation_failover_test.go:93: federation_failover_test.go:93: STEP 13: Check if east-mesh can see services from west-mesh federation_failover_test.go:95: SUCCESS: ratings service from west-mesh successfully imported in east-mesh federation_failover_test.go:104: federation_failover_test.go:104: STEP 14: Send HTTP request to east-mesh, expecting ratings-v1 in east-mesh to receive ratings request and not west-mesh federation_failover_test.go:140: SUCCESS: ratings-v1 in east-mesh received 10 requests federation_failover_test.go:146: SUCCESS: ratings-v1 in west-mesh received no requests federation_failover_test.go:107: federation_failover_test.go:107: STEP 15: Scale Deployment ratings-v1 in east-mesh to zero replicas in order to trigger failover to west-mesh federation_failover_test.go:110: federation_failover_test.go:110: STEP 16: Send HTTP request to east-mesh, expecting ratings-v1 in west-mesh to receive ratings request federation_failover_test.go:162: SUCCESS: ratings-v1 in west-mesh received 10 requests federation_failover_test.go:113: federation_failover_test.go:113: STEP 17: Scale Deployment ratings-v1 in east-mesh back to one replica in order to disable failover again federation_failover_test.go:116: federation_failover_test.go:116: STEP 18: Send HTTP request to east-mesh, expecting ratings-v1 in east-mesh to receive ratings request and not west-mesh federation_failover_test.go:140: SUCCESS: ratings-v1 in east-mesh received 10 requests federation_failover_test.go:146: SUCCESS: ratings-v1 in west-mesh received no requests test.go:76: test.go:81: Test completed in 362.04s (excluding cleanup) federation_common.go:53: federation_common.go:53: Performing cleanup federation_common.go:58: Deleting namespaces: [west-mesh-system bookinfo-ha] federation_common.go:59: Deleting namespaces: [east-mesh-system bookinfo-ha] federation_common.go:53: Cleanup completed in 31.30s --- PASS: TestMultiClusterFederationFailover (393.34s) PASS ok github.com/maistra/maistra-test-tool/pkg/tests/ossm-federation 394.732s DONE 1 tests in 395.442s