[root@docpa125s bin]# subctl verify --verbose --context default/api-di1001-cpaas-test:6443/tadimarvadmin --tocontext default/api-qi1001-cpaas-test:6443/tadimarvadmin --image-override submariner-nettest=quay.io/submariner/nettest:devel ? You have specified disruptive verifications (gateway-failover). Are you sure you want to run them? Yes Performing the following verifications: service-discovery, compliance, connectivity, basic-connectivity, gateway-failover Jul 17 16:26:32.492: Creating kubernetes clients Jul 17 16:26:32.514: Setting new cluster ID "di1001", previous cluster ID was "api-di1001-cpaas-test:6443" Jul 17 16:26:32.542: Setting new cluster ID "qi1001", previous cluster ID was "api-qi1001-cpaas-test:6443" Jul 17 16:26:32.542: Creating submariner clients Jul 17 16:26:32.622: Creating lighthouse clients Running Suite: Submariner E2E suite - /root/.local/bin ====================================================== Random Seed: 1 Will run 34 of 48 specs S ------------------------------ FIPS when FIPS mode is enabled for the active gateway node should use FIPS mode in the libreswan cable driver [compliance] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/compliance/fips.go:36 Jul 17 16:26:32.740: Creating namespace objects with basename "fips-gateway-status" Jul 17 16:26:32.761: Generated namespace "e2e-tests-fips-gateway-status-zql8h" in cluster "di1001" to execute the tests in Jul 17 16:26:32.761: Creating namespace "e2e-tests-fips-gateway-status-zql8h" in cluster "qi1001" Jul 17 16:26:32.802: Find a cluster with FIPS enabled Jul 17 16:26:32.831: Jul 17 16:26:32.831: INFO: No cluster found with FIPS enabled, skipping the test... [SKIPPED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/logging.go:57 @ 07/17/25 16:26:32.831 Jul 17 16:26:32.831: Deleting namespace "e2e-tests-fips-gateway-status-zql8h" on cluster "di1001" Jul 17 16:26:32.839: Deleting namespace "e2e-tests-fips-gateway-status-zql8h" on cluster "qi1001" S [SKIPPED] [0.116 seconds] FIPS when FIPS mode is enabled for the active gateway node [It] should use FIPS mode in the libreswan cable driver [compliance] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/compliance/fips.go:36 [SKIPPED] Jul 17 16:26:32.831: No cluster found with FIPS enabled, skipping the test... In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/logging.go:57 @ 07/17/25 16:26:32.831 ------------------------------ Test Service Discovery Across Clusters when a pod tries to resolve a service in a remote cluster should be able to discover the remote service successfully [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:53 Jul 17 16:26:32.856: Creating namespace objects with basename "discovery" Jul 17 16:26:32.868: Generated namespace "e2e-tests-discovery-qlmgl" in cluster "di1001" to execute the tests in Jul 17 16:26:32.868: Creating namespace "e2e-tests-discovery-qlmgl" in cluster "qi1001" Jul 17 16:26:32.896: Creating an Nginx Deployment on "qi1001" Jul 17 16:26:46.962: Creating a Nginx Service on "qi1001" Jul 17 16:26:47.002: Creating serviceExport nginx-demo.e2e-tests-discovery-qlmgl on "qi1001" Jul 17 16:26:47.017: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-qlmgl on "qi1001" [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:30:47.018 Jul 17 16:30:47.018: Deleting namespace "e2e-tests-discovery-qlmgl" on cluster "di1001" Jul 17 16:30:47.051: Deleting namespace "e2e-tests-discovery-qlmgl" on cluster "qi1001" Jul 17 16:30:47.059: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-qlmgl" on "qi1001" Jul 17 16:30:47.064: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-qlmgl" on "di1001" • [FAILED] [254.212 seconds] Test Service Discovery Across Clusters when a pod tries to resolve a service in a remote cluster [It] should be able to discover the remote service successfully [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:53 [FAILED] Failed to retrieve ServiceExport. ServiceExport Ready condition status is { "type": "Ready", "status": "False", "lastTransitionTime": "2025-07-17T06:30:23Z", "reason": "ExportFailed", "message": "Unable to export: error creating or updating resource: error retrieving \"nginx-demo-e2e-tests-discovery-qlmgl\": Get \"https://api.cpaas-tc1002.gkee.p3.openshiftapps.com:443/apis/multicluster.x-k8s.io/v1alpha1/namespaces/anz-onprem-gnet-set-broker/serviceimports/nginx-demo-e2e-tests-discovery-qlmgl\": read tcp 100.120.74.23:39248-\u003e10.54.254.191:80: read: connection reset by peer - error from a previous attempt: read tcp 100.120.74.23:39236-\u003e10.54.254.191:80: read: connection reset by peer" } Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:30:47.018 ------------------------------ Test Service Discovery Across Clusters when a pod tries to resolve a service which is present locally and in a remote cluster should resolve the local service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:59 Jul 17 16:30:47.068: Creating namespace objects with basename "discovery" Jul 17 16:30:47.088: Generated namespace "e2e-tests-discovery-v8wjf" in cluster "di1001" to execute the tests in Jul 17 16:30:47.088: Creating namespace "e2e-tests-discovery-v8wjf" in cluster "qi1001" Jul 17 16:30:47.117: Creating an Nginx Deployment on "di1001" Jul 17 16:30:52.647: Creating a Nginx Service on "di1001" Jul 17 16:30:52.674: Creating an Nginx Deployment on "qi1001" Jul 17 16:31:13.731: Creating a Nginx Service on "qi1001" Jul 17 16:31:13.753: Creating serviceExport nginx-demo.e2e-tests-discovery-v8wjf on "qi1001" Jul 17 16:31:13.766: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-v8wjf on "qi1001" [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:35:13.777 Jul 17 16:35:13.777: Deleting namespace "e2e-tests-discovery-v8wjf" on cluster "di1001" Jul 17 16:35:13.815: Deleting namespace "e2e-tests-discovery-v8wjf" on cluster "qi1001" Jul 17 16:35:13.832: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-v8wjf" on "qi1001" Jul 17 16:35:13.838: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-v8wjf" on "di1001" • [FAILED] [266.774 seconds] Test Service Discovery Across Clusters when a pod tries to resolve a service which is present locally and in a remote cluster [It] should resolve the local service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:59 [FAILED] Failed to retrieve ServiceExport. ServiceExport Ready condition status is { "type": "Ready", "status": "False", "lastTransitionTime": "2025-07-17T06:34:58Z", "reason": "ExportFailed", "message": "Unable to export: error creating or updating resource: error retrieving \"nginx-demo-e2e-tests-discovery-v8wjf\": Get \"https://api.cpaas-tc1002.gkee.p3.openshiftapps.com:443/apis/multicluster.x-k8s.io/v1alpha1/namespaces/anz-onprem-gnet-set-broker/serviceimports/nginx-demo-e2e-tests-discovery-v8wjf\": read tcp 100.120.74.23:44496-\u003e10.54.254.191:80: read: connection reset by peer - error from a previous attempt: read tcp 100.120.74.23:44466-\u003e10.54.254.191:80: read: connection reset by peer" } Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:35:13.777 ------------------------------ Test Service Discovery Across Clusters when service export is created before the service should resolve the service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:65 Jul 17 16:35:13.843: Creating namespace objects with basename "discovery" Jul 17 16:35:13.860: Generated namespace "e2e-tests-discovery-4fdxd" in cluster "di1001" to execute the tests in Jul 17 16:35:13.860: Creating namespace "e2e-tests-discovery-4fdxd" in cluster "qi1001" Jul 17 16:35:13.908: Creating an Nginx ServiceExport on "qi1001" Jul 17 16:35:13.908: Creating serviceExport nginx-demo.e2e-tests-discovery-4fdxd on "qi1001" Jul 17 16:35:13.926: Creating an Nginx Deployment on "qi1001" Jul 17 16:35:20.460: Creating a Nginx Service on "qi1001" Jul 17 16:35:20.546: Creating a Netshoot Deployment on "di1001" Jul 17 16:35:47.584: Retrieving service nginx-demo.e2e-tests-discovery-4fdxd on "qi1001" Jul 17 16:35:47.588: Retrieving ServiceImport for "nginx-demo" in ns "e2e-tests-discovery-4fdxd" on "di1001" [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:39:47.593 Jul 17 16:39:47.593: Deleting namespace "e2e-tests-discovery-4fdxd" on cluster "di1001" Jul 17 16:39:47.599: Deleting namespace "e2e-tests-discovery-4fdxd" on cluster "qi1001" Jul 17 16:39:47.638: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-4fdxd" on "qi1001" Jul 17 16:39:47.643: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-4fdxd" on "di1001" • [FAILED] [273.804 seconds] Test Service Discovery Across Clusters when service export is created before the service [It] should resolve the service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:65 [FAILED] Failed to retrieve ServiceImport. ServiceImport not found Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:39:47.593 ------------------------------ Test Service Discovery Across Clusters when there are no active pods for a service should not resolve the service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:70 Jul 17 16:39:47.647: Creating namespace objects with basename "discovery" Jul 17 16:39:47.658: Generated namespace "e2e-tests-discovery-68mj6" in cluster "di1001" to execute the tests in Jul 17 16:39:47.658: Creating namespace "e2e-tests-discovery-68mj6" in cluster "qi1001" Jul 17 16:39:47.680: Creating an Nginx Deployment on "qi1001" Jul 17 16:40:00.729: Creating a Nginx Service on "qi1001" Jul 17 16:40:00.772: Creating serviceExport nginx-demo.e2e-tests-discovery-68mj6 on "qi1001" Jul 17 16:40:00.788: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-68mj6 on "qi1001" [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:44:00.81 Jul 17 16:44:00.810: Deleting namespace "e2e-tests-discovery-68mj6" on cluster "di1001" Jul 17 16:44:00.843: Deleting namespace "e2e-tests-discovery-68mj6" on cluster "qi1001" Jul 17 16:44:00.862: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-68mj6" on "qi1001" Jul 17 16:44:00.871: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-68mj6" on "di1001" • [FAILED] [253.229 seconds] Test Service Discovery Across Clusters when there are no active pods for a service [It] should not resolve the service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:70 [FAILED] Failed to retrieve ServiceExport. ServiceExport Ready condition status is { "type": "Ready", "status": "False", "lastTransitionTime": "2025-07-17T06:43:53Z", "reason": "ExportFailed", "message": "Unable to export: error creating or updating resource: error retrieving \"nginx-demo-e2e-tests-discovery-68mj6\": Get \"https://api.cpaas-tc1002.gkee.p3.openshiftapps.com:443/apis/multicluster.x-k8s.io/v1alpha1/namespaces/anz-onprem-gnet-set-broker/serviceimports/nginx-demo-e2e-tests-discovery-68mj6\": read tcp 100.120.74.23:50604-\u003e10.54.254.191:80: read: connection reset by peer - error from a previous attempt: read tcp 100.120.74.23:50584-\u003e10.54.254.191:80: read: connection reset by peer" } Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:44:00.81 ------------------------------ Test Service Discovery Across Clusters when there are active pods for a service in only one cluster should not resolve the service on the cluster without active pods [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:76 Jul 17 16:44:00.876: Creating namespace objects with basename "discovery" Jul 17 16:44:00.898: Generated namespace "e2e-tests-discovery-pbggs" in cluster "di1001" to execute the tests in Jul 17 16:44:00.898: Creating namespace "e2e-tests-discovery-pbggs" in cluster "qi1001" [SKIPPED] in [It] - /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:325 @ 07/17/25 16:44:00.928 Jul 17 16:44:00.928: Deleting namespace "e2e-tests-discovery-pbggs" on cluster "di1001" Jul 17 16:44:00.951: Deleting namespace "e2e-tests-discovery-pbggs" on cluster "qi1001" Jul 17 16:44:00.980: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-pbggs" on "qi1001" Jul 17 16:44:00.997: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-pbggs" on "di1001" S [SKIPPED] [0.124 seconds] Test Service Discovery Across Clusters when there are active pods for a service in only one cluster [It] should not resolve the service on the cluster without active pods [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:76 [SKIPPED] Only two clusters are deployed and hence skipping the test In [It] at: /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:325 @ 07/17/25 16:44:00.928 ------------------------------ Test Service Discovery Across Clusters when a pod tries to resolve a service in a specific remote cluster by its cluster name should resolve the service on the specified cluster [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:82 Jul 17 16:44:01.001: Creating namespace objects with basename "discovery" Jul 17 16:44:01.016: Generated namespace "e2e-tests-discovery-l2tsd" in cluster "di1001" to execute the tests in Jul 17 16:44:01.016: Creating namespace "e2e-tests-discovery-l2tsd" in cluster "qi1001" Jul 17 16:44:01.042: Creating an Nginx Deployment on "di1001" Jul 17 16:44:15.072: Creating a Nginx Service on "di1001" Jul 17 16:44:15.112: Creating serviceExport nginx-demo.e2e-tests-discovery-l2tsd on "di1001" Jul 17 16:44:15.122: Creating an Nginx Deployment on "qi1001" Jul 17 16:44:20.159: Creating a Nginx Service on "qi1001" Jul 17 16:44:20.206: Creating serviceExport nginx-demo.e2e-tests-discovery-l2tsd on "qi1001" Jul 17 16:44:20.235: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-l2tsd on "qi1001" [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:48:20.24 Jul 17 16:48:20.240: Deleting namespace "e2e-tests-discovery-l2tsd" on cluster "di1001" Jul 17 16:48:20.288: Deleting namespace "e2e-tests-discovery-l2tsd" on cluster "qi1001" Jul 17 16:48:20.296: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-l2tsd" on "qi1001" Jul 17 16:48:20.300: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-l2tsd" on "di1001" • [FAILED] [259.303 seconds] Test Service Discovery Across Clusters when a pod tries to resolve a service in a specific remote cluster by its cluster name [It] should resolve the service on the specified cluster [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:82 [FAILED] Failed to retrieve ServiceExport. ServiceExport Ready condition status is { "type": "Ready", "status": "False", "lastTransitionTime": "2025-07-17T06:47:31Z", "reason": "ExportFailed", "message": "Unable to export: error creating or updating resource: error retrieving \"nginx-demo-e2e-tests-discovery-l2tsd\": Get \"https://api.cpaas-tc1002.gkee.p3.openshiftapps.com:443/apis/multicluster.x-k8s.io/v1alpha1/namespaces/anz-onprem-gnet-set-broker/serviceimports/nginx-demo-e2e-tests-discovery-l2tsd\": read tcp 100.120.74.23:48112-\u003e10.54.254.191:80: read: connection reset by peer - error from a previous attempt: read tcp 100.120.74.23:48106-\u003e10.54.254.191:80: read: connection reset by peer" } Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 @ 07/17/25 16:48:20.24 ------------------------------ Test Service Discovery Across Clusters when a pod tries to resolve a service multiple times should resolve the service from both the clusters in a round robin fashion [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:88 Jul 17 16:48:20.304: Creating namespace objects with basename "discovery" Jul 17 16:48:20.315: Generated namespace "e2e-tests-discovery-t6cqx" in cluster "di1001" to execute the tests in Jul 17 16:48:20.315: Creating namespace "e2e-tests-discovery-t6cqx" in cluster "qi1001" [SKIPPED] in [It] - /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:458 @ 07/17/25 16:48:20.338 Jul 17 16:48:20.338: Deleting namespace "e2e-tests-discovery-t6cqx" on cluster "di1001" Jul 17 16:48:20.346: Deleting namespace "e2e-tests-discovery-t6cqx" on cluster "qi1001" Jul 17 16:48:20.353: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-t6cqx" on "qi1001" Jul 17 16:48:20.360: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-t6cqx" on "di1001" S [SKIPPED] [0.062 seconds] Test Service Discovery Across Clusters when a pod tries to resolve a service multiple times [It] should resolve the service from both the clusters in a round robin fashion [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:88 [SKIPPED] Only two clusters are deployed and hence skipping the test In [It] at: /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:458 @ 07/17/25 16:48:20.338 ------------------------------ Test Service Discovery Across Clusters when one of the clusters with a service is not healthy should not resolve that cluster's service IP [service-discovery] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:113 Jul 17 16:48:20.366: Creating namespace objects with basename "discovery" Jul 17 16:48:20.384: Generated namespace "e2e-tests-discovery-zhct8" in cluster "di1001" to execute the tests in Jul 17 16:48:20.384: Creating namespace "e2e-tests-discovery-zhct8" in cluster "qi1001" [SKIPPED] in [BeforeEach] - /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:98 @ 07/17/25 16:48:20.399 Jul 17 16:48:20.399: Deleting namespace "e2e-tests-discovery-zhct8" on cluster "di1001" Jul 17 16:48:20.409: Deleting namespace "e2e-tests-discovery-zhct8" on cluster "qi1001" Jul 17 16:48:20.419: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-zhct8" on "qi1001" Jul 17 16:48:20.425: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-zhct8" on "di1001" S [SKIPPED] [0.062 seconds] Test Service Discovery Across Clusters when one of the clusters with a service is not healthy [BeforeEach] should not resolve that cluster's service IP [service-discovery] [BeforeEach] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:96 [It] /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:113 [SKIPPED] Only two clusters are deployed and hence skipping the test In [BeforeEach] at: /remote-source/app/vendor/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:98 @ 07/17/25 16:48:20.399 ------------------------------ Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway should have sent the expected data from the pod to the other pod [dataplane, globalnet, basic] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Jul 17 16:48:20.429: Creating namespace objects with basename "dataplane-gn-conn-nd" Jul 17 16:48:20.442: Generated namespace "e2e-tests-dataplane-gn-conn-nd-z7bkt" in cluster "di1001" to execute the tests in Jul 17 16:48:20.442: Creating namespace "e2e-tests-dataplane-gn-conn-nd-z7bkt" in cluster "qi1001" Jul 17 16:48:20.490: Creating a listener pod in cluster "qi1001", which will wait for a handshake over TCP Jul 17 16:48:32.069: Pointing a ClusterIP service to the listener pod in cluster "qi1001" Jul 17 16:48:32.665: Creating a connector pod in cluster "di1001", which will attempt the specific UUID handshake over TCP Jul 17 16:48:37.217: Jul 17 16:48:37.217: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says bc014aec-e775-48fa-9957-709fa4a439b2; done | for i in $(seq 2); do if nc -v 242.1.255.253 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-z7bkt PodName:customxf4t7 ContainerName:connector-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} Jul 17 16:51:37.878: Connector pod is scheduled on node "docpa126s" Jul 17 16:51:37.878: Waiting for the listener pod "tcp-check-listenerfsmzx" on node "qocpa101s" to exit, returning what listener sent [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 16:55:37.879 Jul 17 16:55:37.879: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-z7bkt" on cluster "di1001" Jul 17 16:55:37.915: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-z7bkt" on cluster "qi1001" • [FAILED] [437.500 seconds] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet, basic] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 [FAILED] Failed to await pod "tcp-check-listenerfsmzx" finished. Pod status is Running Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 16:55:37.879 ------------------------------ Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is not on a gateway should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Jul 17 16:55:37.929: Creating namespace objects with basename "dataplane-gn-conn-nd" Jul 17 16:55:37.940: Generated namespace "e2e-tests-dataplane-gn-conn-nd-dxh2c" in cluster "di1001" to execute the tests in Jul 17 16:55:37.940: Creating namespace "e2e-tests-dataplane-gn-conn-nd-dxh2c" in cluster "qi1001" Jul 17 16:55:37.982: Creating a listener pod in cluster "qi1001", which will wait for a handshake over TCP Jul 17 16:55:43.050: Pointing a ClusterIP service to the listener pod in cluster "qi1001" Jul 17 16:55:43.752: Creating a connector pod in cluster "di1001", which will attempt the specific UUID handshake over TCP Jul 17 16:55:56.318: Jul 17 16:55:56.318: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says 512fb7a1-8048-4382-aef2-20e7303773b0; done | for i in $(seq 2); do if nc -v 242.1.255.252 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-dxh2c PodName:customqxlhb ContainerName:connector-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} Jul 17 16:58:56.636: Connector pod is scheduled on node "docpa106s" Jul 17 16:58:56.636: Waiting for the listener pod "tcp-check-listenerjmxjc" on node "qocpa149s" to exit, returning what listener sent [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:02:56.643 Jul 17 17:02:56.643: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-dxh2c" on cluster "di1001" Jul 17 17:02:56.682: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-dxh2c" on cluster "qi1001" • [FAILED] [438.761 seconds] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 [FAILED] Failed to await pod "tcp-check-listenerjmxjc" finished. Pod status is Running Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:02:56.643 ------------------------------ Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway should have sent the expected data from the pod to the other pod [dataplane, globalnet, basic] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Jul 17 17:02:56.691: Creating namespace objects with basename "dataplane-gn-conn-nd" Jul 17 17:02:56.704: Generated namespace "e2e-tests-dataplane-gn-conn-nd-nhr7j" in cluster "di1001" to execute the tests in Jul 17 17:02:56.704: Creating namespace "e2e-tests-dataplane-gn-conn-nd-nhr7j" in cluster "qi1001" Jul 17 17:02:56.723: Creating a listener pod in cluster "qi1001", which will wait for a handshake over TCP Jul 17 17:03:12.286: Pointing a ClusterIP service to the listener pod in cluster "qi1001" Jul 17 17:03:12.869: Creating a connector pod in cluster "di1001", which will attempt the specific UUID handshake over TCP Jul 17 17:03:17.429: Jul 17 17:03:17.429: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says 76f3b55c-d3f3-4a16-ba28-9306028df5f6; done | for i in $(seq 2); do if nc -v 242.1.255.253 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-nhr7j PodName:customvnzpw ContainerName:connector-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} Jul 17 17:06:17.703: Connector pod is scheduled on node "docpa106s" Jul 17 17:06:17.703: Waiting for the listener pod "tcp-check-listenerts4k6" on node "qocpa137s" to exit, returning what listener sent [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:10:17.704 Jul 17 17:10:17.704: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-nhr7j" on cluster "di1001" Jul 17 17:10:17.738: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-nhr7j" on cluster "qi1001" • [FAILED] [441.056 seconds] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet, basic] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 [FAILED] Failed to await pod "tcp-check-listenerts4k6" finished. Pod status is Running Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:10:17.704 ------------------------------ Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Jul 17 17:10:17.747: Creating namespace objects with basename "dataplane-gn-conn-nd" Jul 17 17:10:17.763: Generated namespace "e2e-tests-dataplane-gn-conn-nd-pn4mp" in cluster "di1001" to execute the tests in Jul 17 17:10:17.763: Creating namespace "e2e-tests-dataplane-gn-conn-nd-pn4mp" in cluster "qi1001" Jul 17 17:10:18.311: Creating a listener pod in cluster "qi1001", which will wait for a handshake over TCP Jul 17 17:10:31.381: Pointing a ClusterIP service to the listener pod in cluster "qi1001" Jul 17 17:10:31.967: Creating a connector pod in cluster "di1001", which will attempt the specific UUID handshake over TCP Jul 17 17:10:43.542: Jul 17 17:10:43.542: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says 4c0724c7-846b-4fdf-9142-7a3d2f2523b2; done | for i in $(seq 2); do if nc -v 242.1.255.253 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-pn4mp PodName:customphb8f ContainerName:connector-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} Jul 17 17:13:43.848: Connector pod is scheduled on node "docpa103s" Jul 17 17:13:43.848: Waiting for the listener pod "tcp-check-listenerlthbs" on node "qocpa123s" to exit, returning what listener sent [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:17:43.848 Jul 17 17:17:43.848: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-pn4mp" on cluster "di1001" Jul 17 17:17:43.881: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-pn4mp" on cluster "qi1001" • [FAILED] [446.144 seconds] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 [FAILED] Failed to await pod "tcp-check-listenerlthbs" finished. Pod status is Running Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:17:43.848 ------------------------------ Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Jul 17 17:17:43.891: Creating namespace objects with basename "dataplane-gn-conn-nd" Jul 17 17:17:43.904: Generated namespace "e2e-tests-dataplane-gn-conn-nd-cpnzd" in cluster "di1001" to execute the tests in Jul 17 17:17:43.904: Creating namespace "e2e-tests-dataplane-gn-conn-nd-cpnzd" in cluster "qi1001" Jul 17 17:17:44.450: Creating a listener pod in cluster "qi1001", which will wait for a handshake over TCP Jul 17 17:17:47.519: Pointing a ClusterIP service to the listener pod in cluster "qi1001" Jul 17 17:17:48.132: Creating a connector pod in cluster "di1001", which will attempt the specific UUID handshake over TCP Jul 17 17:17:52.696: Jul 17 17:17:52.696: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says 00de210a-beff-465e-9d3d-bbf01c9708f1; done | for i in $(seq 2); do if nc -v 242.1.255.252 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-cpnzd PodName:custommskfh ContainerName:connector-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} Jul 17 17:20:52.997: Connector pod is scheduled on node "docpa106s" Jul 17 17:20:52.997: Waiting for the listener pod "tcp-check-listenerwfwk4" on node "qocpa137s" to exit, returning what listener sent [FAILED] in [It] - /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:24:53.001 Jul 17 17:24:53.001: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-cpnzd" on cluster "di1001" Jul 17 17:24:53.047: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-cpnzd" on cluster "qi1001" • [FAILED] [429.165 seconds] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 [FAILED] Failed to await pod "tcp-check-listenerwfwk4" finished. Pod status is Running Unexpected error: : context deadline exceeded {} occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 @ 07/17/25 17:24:53.001 ------------------------------ Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP pod selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Jul 17 17:24:53.055: Creating namespace objects with basename "dataplane-gn-conn-nd" Jul 17 17:24:53.072: Generated namespace "e2e-tests-dataplane-gn-conn-nd-q24cf" in cluster "di1001" to execute the tests in Jul 17 17:24:53.072: Creating namespace "e2e-tests-dataplane-gn-conn-nd-q24cf" in cluster "qi1001" Jul 17 17:24:53.675: Creating a listener pod in cluster "qi1001", which will wait for a handshake over TCP Jul 17 17:24:57.753: Pointing a ClusterIP service to the listener pod in cluster "qi1001" Jul 17 17:24:58.308: Creating a connector pod in cluster "di1001", which will attempt the specific UUID handshake over TCP Jul 17 17:25:10.904: Jul 17 17:25:10.904: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says e89bd3e7-b399-4a18-a12d-f900387d6f18; done | for i in $(seq 2); do if nc -v 242.1.255.253 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-q24cf PodName:customqzcmb ContainerName:connector-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} [TIMEDOUT] in [It] - /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 @ 07/17/25 17:26:32.74 Jul 17 17:26:32.741: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-q24cf" on cluster "di1001" Jul 17 17:26:32.749: Deleting namespace "e2e-tests-dataplane-gn-conn-nd-q24cf" on cluster "qi1001" • [TIMEDOUT] [99.729 seconds] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP pod selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 [TIMEDOUT] A suite timeout occurred In [It] at: /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 @ 07/17/25 17:26:32.74 This is the Progress Report generated when the suite timeout occurred: Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP pod selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway should have sent the expected data from the pod to the other pod (Spec Runtime: 1m39.685s) /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 In [It] (Node Runtime: 1m39.655s) /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Spec Goroutine goroutine 5991 [select] k8s.io/client-go/tools/remotecommand.(*spdyStreamExecutor).StreamWithContext(0x0?, {0x22d27f0, 0x34c81e0}, {{0x0, 0x0}, {0x22ae5e0, 0xc000e80c90}, {0x22ae5e0, 0xc000e80cc0}, 0x0, ...}) /remote-source/app/vendor/k8s.io/client-go/tools/remotecommand/spdy.go:163 github.com/submariner-io/shipyard/test/e2e/framework.execute({0x22d27f0, 0x34c81e0}, {0x1fce875?, 0x1fcdea1?}, 0xc00006b8a0?, 0x46cb3d?, {0x0, 0x0}, {0x22ae5e0, 0xc000e80c90}, ...) /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/exec.go:107 github.com/submariner-io/shipyard/test/e2e/framework.(*Framework).ExecWithOptions(0x2066d96?, {0x22d27f0, 0x34c81e0}, 0xc000e0c960, 0x0) /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/exec.go:83 github.com/submariner-io/submariner/test/e2e/framework.verifyGlobalnetDatapathConnectivity.func1(0xc0004c53b0, 0xc0004c5c70) /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/framework/dataplane.go:169 github.com/submariner-io/submariner/test/e2e/framework.verifyGlobalnetDatapathConnectivity({0xc0007801e0, 0x0, 0x3c, 0x2, 0x0, 0x2, 0x1, 0x2, 0x2}, {0x1, ...}) /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/framework/dataplane.go:206 github.com/submariner-io/submariner/test/e2e/framework.VerifyDatapathConnectivity({0xc0007801e0, 0x0, 0x0, 0x0, 0x0, 0x2, 0x1, 0x2, 0x2}, {0x1, ...}) /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/framework/dataplane.go:73 > github.com/submariner-io/submariner/test/e2e/dataplane.init.func2.2.1() /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:64 github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0xc000304080?, 0xc000649fd0?}) /remote-source/app/vendor/github.com/onsi/ginkgo/v2/internal/node.go:475 github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3() /remote-source/app/vendor/github.com/onsi/ginkgo/v2/internal/suite.go:894 github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 1 /remote-source/app/vendor/github.com/onsi/ginkgo/v2/internal/suite.go:881 ------------------------------ SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS Summarizing 11 Failures: [FAIL] Test Service Discovery Across Clusters when a pod tries to resolve a service in a remote cluster [It] should be able to discover the remote service successfully [service-discovery] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 [FAIL] Test Service Discovery Across Clusters when a pod tries to resolve a service which is present locally and in a remote cluster [It] should resolve the local service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 [FAIL] Test Service Discovery Across Clusters when service export is created before the service [It] should resolve the service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 [FAIL] Test Service Discovery Across Clusters when there are no active pods for a service [It] should not resolve the service [service-discovery] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 [FAIL] Test Service Discovery Across Clusters when a pod tries to resolve a service in a specific remote cluster by its cluster name [It] should resolve the service on the specified cluster [service-discovery] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562 [FAIL] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet, basic] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 [FAIL] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 [FAIL] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet, basic] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 [FAIL] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 [FAIL] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/shipyard/test/e2e/framework/network_pods.go:195 [TIMEDOUT] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP pod selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] /remote-source/app/vendor/github.com/submariner-io/submariner/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 Ran 11 of 48 Specs in 3600.046 seconds FAIL! - Suite Timeout Elapsed -- 0 Passed | 11 Failed | 0 Pending | 37 Skipped subctl version: v0.20.1 [root@docpa125s bin]# [root@docpa125s bin]#