exec: [go test -json -tags=integ -timeout 180m ./tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=true -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 4134399 2025-08-01T14:11:46.826136Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:11:46.826197Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:11:46.860335Z info tf === Test Framework Settings === 2025-08-01T14:11:46.860473Z info tf TestID: telemetry_tracing_zipkin RunID: a10aa6f8-af8b-4196-b1ad-cb82b3de6758 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-01T14:11:46.860480Z info tf =============================== 2025-08-01T14:11:46.860880Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-a10aa6 2025-08-01T14:11:46.860973Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-01T14:11:46.860988Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-01T14:11:46.861006Z info tf Using KubeConfigs: [/root/upi/cO/auth/kubeconfig]. 2025-08-01T14:11:46.861014Z info tf === BEGIN: Building clusters === 2025-08-01T14:11:46.864474Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cO/auth/kubeconfig 2025-08-01T14:11:46.864483Z info tf === DONE: Building clusters === 2025-08-01T14:11:46.864504Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-08-01T14:11:46.865948Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:11:46.866036Z info tf === Istio Component Config === 2025-08-01T14:11:46.866273Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-01T14:11:46.866831Z info tf ================================ 2025-08-01T14:11:46.869567Z info tf skipping deployment as specified in the config 2025-08-01T14:11:46.869585Z info tf === SUCCEEDED: Deploy Istio in 4.706489ms [Suite=telemetry_tracing_zipkin]=== 2025-08-01T14:11:46.869592Z info tf === BEGIN: Create namespace echo === 2025-08-01T14:11:46.890758Z info tf === SUCCEEDED: Create namespace echo in 21.155868ms === 2025-08-01T14:11:46.924797Z info tf === BEGIN: Deploy echo instances === 2025-08-01T14:11:47.013910Z info klog Waiting for caches to sync controller=echo 2025-08-01T14:11:47.017043Z info klog Waiting for caches to sync controller=echo 2025-08-01T14:11:47.114654Z info klog Caches are synced controller=echo 2025-08-01T14:11:47.118201Z info klog Caches are synced controller=echo 2025-08-01T14:11:56.622774Z info tf === SUCCEEDED: Deploy echo instances in 9.697975979s === 2025-08-01T14:11:56.710871Z info tf Checking pods ready... 2025-08-01T14:11:56.710904Z info tf Checking pods ready... 2025-08-01T14:11:56.714727Z info tf Failed retrieving pods: no matching pod found for selectors: [app=zipkin] 2025-08-01T14:11:56.915610Z info tf Checking pods ready... 2025-08-01T14:11:56.915719Z info tf Checking pods ready... 2025-08-01T14:11:56.920240Z info tf [ 0] zipkin-5c6fbc9d85-vvtkt Pending (Pending) 2025-08-01T14:11:57.320352Z info tf Checking pods ready... 2025-08-01T14:11:57.320423Z info tf Checking pods ready... 2025-08-01T14:11:57.325480Z info tf [ 0] zipkin-5c6fbc9d85-vvtkt Pending (Pending) 2025-08-01T14:11:58.125648Z info tf Checking pods ready... 2025-08-01T14:11:58.125721Z info tf Checking pods ready... 2025-08-01T14:11:58.129748Z info tf [ 0] zipkin-5c6fbc9d85-vvtkt Pending (Pending) 2025-08-01T14:11:59.730800Z info tf Checking pods ready... 2025-08-01T14:11:59.730875Z info tf Checking pods ready... 2025-08-01T14:11:59.739683Z info tf [ 0] zipkin-5c6fbc9d85-vvtkt Running (container not ready: 'zipkin') 2025-08-01T14:12:02.939773Z info tf Checking pods ready... 2025-08-01T14:12:02.939868Z info tf Checking pods ready... 2025-08-01T14:12:02.947841Z info tf [ 0] zipkin-5c6fbc9d85-vvtkt Running (container not ready: 'zipkin') 2025-08-01T14:12:06.148610Z info tf Checking pods ready... 2025-08-01T14:12:06.148699Z info tf Checking pods ready... 2025-08-01T14:12:06.152474Z info tf [ 0] zipkin-5c6fbc9d85-vvtkt Running (Ready) 2025-08-01T14:12:06.216921Z info tf === DONE: Setup: 'telemetry_tracing_zipkin' (19.352402285s) === 2025-08-01T14:12:06.216967Z info tf === BEGIN: Test Run: 'telemetry_tracing_zipkin' === === RUN TestClientTracing/primary-0 2025-08-01T14:12:06.217350Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0]' === client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:43: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:14:36.218429Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0] (2m30.001064025s)' === 2025-08-01T14:14:36.218639Z error tf === Dumping Namespace echo State for TestClientTracing/primary-0... 2025-08-01T14:14:36.218638Z error tf === Dumping Istio Deployment State for TestClientTracing/primary-0... 2025-08-01T14:14:36.375563Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63537a53b2-a186-48b4-9152-25f5a4ae4005 2025-08-01T14:14:36.375700Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1567s 2025-08-01T14:14:36.376325Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:36.464301Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6363a59f29-53da-4f9b-8cae-a1043ab8aad4 2025-08-01T14:14:36.464456Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2454s 2025-08-01T14:14:36.465053Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:36.557039Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/637f8307b6-cd85-4d66-b630-a3f5a52e04b3 2025-08-01T14:14:36.557227Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3382s 2025-08-01T14:14:36.559184Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestClientTracing/primary-0 (150.74s) FAIL tests/integration/telemetry/tracing/zipkin.TestClientTracing/primary-0 (150.74s) === RUN TestClientTracing 2025-08-01T14:12:06.217208Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing]' === 2025-08-01T14:14:36.958810Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing] (2m30.741592037s)' === 2025-08-01T14:14:36.958952Z error tf === Dumping Namespace echo State for TestClientTracing... 2025-08-01T14:14:36.958951Z error tf === Dumping Istio Deployment State for TestClientTracing... 2025-08-01T14:14:37.335061Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/633f8a6ffa-135e-405f-abaa-636fa4de31ff 2025-08-01T14:14:37.335302Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3761s 2025-08-01T14:14:37.335889Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:37.438427Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63dc3008dc-6d67-4ded-b598-8b78ee00cebd 2025-08-01T14:14:37.438637Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4794s 2025-08-01T14:14:37.439464Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:37.538752Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63859214fc-f5b2-4d95-b410-dd2abd0c2725 2025-08-01T14:14:37.538976Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5797s 2025-08-01T14:14:37.539594Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestClientTracing (151.46s) FAIL tests/integration/telemetry/tracing/zipkin.TestClientTracing (151.46s) === RUN TestServerTracing/primary-0 2025-08-01T14:14:37.673729Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:17:07.675034Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.001293499s)' === 2025-08-01T14:17:07.675219Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-08-01T14:17:07.675218Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-08-01T14:17:08.038646Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/639c8f4cc5-6b7a-4c05-a8e3-876890b98e08 2025-08-01T14:17:08.038803Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3630s 2025-08-01T14:17:08.039690Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:08.191218Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63b3ffb3cb-54f7-4de5-8ec8-4fec42d17e86 2025-08-01T14:17:08.191390Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5155s 2025-08-01T14:17:08.192210Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:08.354197Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63237c6e02-04db-46d1-8e33-e1f0803598ec 2025-08-01T14:17:08.354565Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6787s 2025-08-01T14:17:08.355108Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing/primary-0 (151.12s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing/primary-0 (151.12s) === RUN TestServerTracing 2025-08-01T14:14:37.673581Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-08-01T14:17:08.798270Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m31.124684067s)' === 2025-08-01T14:17:08.798388Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-08-01T14:17:08.798405Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-08-01T14:17:09.105817Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/630f050b10-c5f5-4b97-8286-f7418469aec3 2025-08-01T14:17:09.106180Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3075s 2025-08-01T14:17:09.106921Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:09.203665Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6377920fb9-194d-4b45-9403-5f6ff5023af3 2025-08-01T14:17:09.203775Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4051s 2025-08-01T14:17:09.205959Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:09.298355Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6348eebaf8-0ee9-4f3d-87e6-104bb34fedb5 2025-08-01T14:17:09.298609Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4999s 2025-08-01T14:17:09.300770Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing (152.18s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing (152.18s) 2025-08-01T14:17:09.851878Z info tf === FAILED: Test Run: 'telemetry_tracing_zipkin' (exitCode: 1) === 2025-08-01T14:17:09.851935Z info tf === Suite "telemetry_tracing_zipkin" run time: 5m22.987436111s === 2025-08-01T14:17:09.851951Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/trace.yaml 2025-08-01T14:17:09.852115Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:17:09.852129Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:17:09.976697Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63f395e3f6-e1ac-49b2-b854-bf8796e1a851 2025-08-01T14:17:09.977028Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1247s 2025-08-01T14:17:09.977778Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:10.072772Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6387c3e0ef-6470-4c4c-93c5-afafd0ead299 2025-08-01T14:17:10.072936Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2205s 2025-08-01T14:17:10.073807Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:10.164636Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6310019dfa-c1eb-41f8-8026-930622aad8ca 2025-08-01T14:17:10.164805Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3124s 2025-08-01T14:17:10.165476Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:10.944834Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:17:10.944996Z info tf === SUCCEEDED: Cleanup Istio in 155.22µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin DONE 4 tests, 4 failures in 336.919s exec: [go test -json -test.run=^TestClientTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=true -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 4136035 2025-08-01T14:17:23.253721Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:17:23.253802Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:17:23.288922Z info tf === Test Framework Settings === 2025-08-01T14:17:23.289005Z info tf TestID: telemetry_tracing_zipkin RunID: d807e5db-1f74-4422-9723-48254ab8227d NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-01T14:17:23.289012Z info tf =============================== 2025-08-01T14:17:23.289269Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-d807e5 2025-08-01T14:17:23.289383Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-01T14:17:23.289399Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-01T14:17:23.289426Z info tf Using KubeConfigs: [/root/upi/cO/auth/kubeconfig]. 2025-08-01T14:17:23.289445Z info tf === BEGIN: Building clusters === 2025-08-01T14:17:23.293998Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cO/auth/kubeconfig 2025-08-01T14:17:23.294017Z info tf === DONE: Building clusters === 2025-08-01T14:17:23.294048Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-08-01T14:17:23.294084Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:17:23.294095Z info tf === Istio Component Config === 2025-08-01T14:17:23.294146Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-01T14:17:23.294155Z info tf ================================ 2025-08-01T14:17:23.297090Z info tf skipping deployment as specified in the config 2025-08-01T14:17:23.297106Z info tf === SUCCEEDED: Deploy Istio in 3.017832ms [Suite=telemetry_tracing_zipkin]=== 2025-08-01T14:17:23.297115Z info tf === BEGIN: Create namespace echo === 2025-08-01T14:17:23.313604Z info tf === SUCCEEDED: Create namespace echo in 16.471545ms === 2025-08-01T14:17:23.357138Z info tf === BEGIN: Deploy echo instances === 2025-08-01T14:17:23.363483Z error tf === FAILED: Deploy echo instances === 2025-08-01T14:17:23.363502Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-d807e5/_suite_context/Service.680138941.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-08-01T14:17:23.363535Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:17:23.363563Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:17:23.458773Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-01T14:17:23.463127Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-01T14:17:23.502072Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:18Z/65a3889123-e8ca-4984-a006-48d4115712da 2025-08-01T14:17:23.503096Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1393s 2025-08-01T14:17:23.503787Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:23.601489Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:18Z/6533a7b5ee-07a0-4118-a9ea-52dd4ed5787a 2025-08-01T14:17:23.601620Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2378s 2025-08-01T14:17:23.602341Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:23.717506Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:18Z/6530c2fbe8-d3f4-44d1-8cb7-e0c46fe30dc9 2025-08-01T14:17:23.717675Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3538s 2025-08-01T14:17:23.718451Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:24.414378Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-6bb9d855c5-rcsdk/istio-proxy for: pods "client-cluster-0-v1-6bb9d855c5-rcsdk" not found 2025-08-01T14:17:24.417551Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-6bb9d855c5-rcsdk/istio-validation for: pods "client-cluster-0-v1-6bb9d855c5-rcsdk" not found 2025-08-01T14:17:24.419750Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-8b87894f7-6ttjh/app for: pods "server-v1-8b87894f7-6ttjh" not found 2025-08-01T14:17:24.422471Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-8b87894f7-6ttjh/istio-proxy for: pods "server-v1-8b87894f7-6ttjh" not found 2025-08-01T14:17:24.425130Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-8b87894f7-6ttjh/istio-validation for: pods "server-v1-8b87894f7-6ttjh" not found 2025-08-01T14:17:24.427992Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-d807e5/_suite_context/Service.680138941.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-08-01T14:17:24.428003Z info tf === FAILED: Setup: 'telemetry_tracing_zipkin' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-d807e5/_suite_context/Service.680138941.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-08-01T14:17:24.428009Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-d807e5/_suite_context/Service.680138941.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-08-01T14:17:24.428037Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:17:24.428113Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:17:24.437066Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:17:24.437177Z info tf === SUCCEEDED: Cleanup Istio in 100.966µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin exec: [go test -json -test.run=^TestServerTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=true -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 4137494 2025-08-01T14:17:38.515163Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:17:38.515219Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:17:38.546695Z info tf === Test Framework Settings === 2025-08-01T14:17:38.546808Z info tf TestID: telemetry_tracing_zipkin RunID: a6de3883-860e-44b1-90b9-778e8e8b13d1 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-01T14:17:38.546817Z info tf =============================== 2025-08-01T14:17:38.547051Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-a6de38 2025-08-01T14:17:38.547156Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-01T14:17:38.547175Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-01T14:17:38.547199Z info tf Using KubeConfigs: [/root/upi/cO/auth/kubeconfig]. 2025-08-01T14:17:38.547210Z info tf === BEGIN: Building clusters === 2025-08-01T14:17:38.550774Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cO/auth/kubeconfig 2025-08-01T14:17:38.550782Z info tf === DONE: Building clusters === 2025-08-01T14:17:38.550815Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-08-01T14:17:38.550851Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:17:38.550862Z info tf === Istio Component Config === 2025-08-01T14:17:38.550920Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-01T14:17:38.550929Z info tf ================================ 2025-08-01T14:17:38.553477Z info tf skipping deployment as specified in the config 2025-08-01T14:17:38.553498Z info tf === SUCCEEDED: Deploy Istio in 2.639927ms [Suite=telemetry_tracing_zipkin]=== 2025-08-01T14:17:38.553509Z info tf === BEGIN: Create namespace echo === 2025-08-01T14:17:38.601487Z info tf === SUCCEEDED: Create namespace echo in 47.965281ms === 2025-08-01T14:17:38.647061Z info tf === BEGIN: Deploy echo instances === 2025-08-01T14:17:38.747877Z info klog Waiting for caches to sync controller=echo 2025-08-01T14:17:38.749527Z info klog Waiting for caches to sync controller=echo 2025-08-01T14:17:38.848839Z info klog Caches are synced controller=echo 2025-08-01T14:17:38.850427Z info klog Caches are synced controller=echo 2025-08-01T14:17:48.357864Z info tf === SUCCEEDED: Deploy echo instances in 9.710787265s === 2025-08-01T14:17:48.401017Z info tf Checking pods ready... 2025-08-01T14:17:48.401065Z info tf Checking pods ready... 2025-08-01T14:17:48.406227Z info tf [ 0] zipkin-5c6fbc9d85-9ftpj Pending (Pending) 2025-08-01T14:17:48.607118Z info tf Checking pods ready... 2025-08-01T14:17:48.607186Z info tf Checking pods ready... 2025-08-01T14:17:48.611588Z info tf [ 0] zipkin-5c6fbc9d85-9ftpj Pending (Pending) 2025-08-01T14:17:49.012627Z info tf Checking pods ready... 2025-08-01T14:17:49.012736Z info tf Checking pods ready... 2025-08-01T14:17:49.018583Z info tf [ 0] zipkin-5c6fbc9d85-9ftpj Pending (Pending) 2025-08-01T14:17:49.819617Z info tf Checking pods ready... 2025-08-01T14:17:49.819675Z info tf Checking pods ready... 2025-08-01T14:17:49.824632Z info tf [ 0] zipkin-5c6fbc9d85-9ftpj Pending (Pending) 2025-08-01T14:17:51.424690Z info tf Checking pods ready... 2025-08-01T14:17:51.424759Z info tf Checking pods ready... 2025-08-01T14:17:51.429377Z info tf [ 0] zipkin-5c6fbc9d85-9ftpj Running (container not ready: 'zipkin') 2025-08-01T14:17:54.630227Z info tf Checking pods ready... 2025-08-01T14:17:54.630350Z info tf Checking pods ready... 2025-08-01T14:17:54.635096Z info tf [ 0] zipkin-5c6fbc9d85-9ftpj Running (container not ready: 'zipkin') 2025-08-01T14:17:57.835889Z info tf Checking pods ready... 2025-08-01T14:17:57.835966Z info tf Checking pods ready... 2025-08-01T14:17:57.841345Z info tf [ 0] zipkin-5c6fbc9d85-9ftpj Running (Ready) 2025-08-01T14:17:57.914489Z info tf === DONE: Setup: 'telemetry_tracing_zipkin' (19.363656265s) === 2025-08-01T14:17:57.914532Z info tf === BEGIN: Test Run: 'telemetry_tracing_zipkin' === === RUN TestServerTracing/primary-0 2025-08-01T14:17:57.915132Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:20:27.916433Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.001290686s)' === 2025-08-01T14:20:27.916621Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-08-01T14:20:27.916621Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-08-01T14:20:28.043429Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/715a9dc70c-f3d9-4f76-801d-58bac01acfe4 2025-08-01T14:20:28.043670Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1267s 2025-08-01T14:20:28.044252Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:28.137613Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/71e5b33997-5f4c-41e7-98b8-58e7e027278d 2025-08-01T14:20:28.137700Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2207s 2025-08-01T14:20:28.138323Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:28.220865Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/718b810261-a217-431f-969e-3b4b0dab22bf 2025-08-01T14:20:28.221085Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3041s 2025-08-01T14:20:28.221648Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing/primary-0 (150.64s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing/primary-0 (re-run 1) (150.64s) === RUN TestServerTracing 2025-08-01T14:17:57.914917Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-08-01T14:20:28.551574Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.63664712s)' === 2025-08-01T14:20:28.551705Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-08-01T14:20:28.551721Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-08-01T14:20:28.839533Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/714cd265d9-4c04-4a58-ac01-0aa277fa7578 2025-08-01T14:20:28.839749Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2877s 2025-08-01T14:20:28.841615Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:28.998680Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/71fc40688d-3932-4c2a-8e2f-dabf2708df3d 2025-08-01T14:20:28.998950Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4469s 2025-08-01T14:20:28.999510Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:29.072220Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/7164d090a2-0191-40dd-b1fc-45e712071e7b 2025-08-01T14:20:29.072372Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5203s 2025-08-01T14:20:29.073112Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing (151.41s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing (re-run 1) (151.41s) 2025-08-01T14:20:29.320546Z info tf === FAILED: Test Run: 'telemetry_tracing_zipkin' (exitCode: 1) === 2025-08-01T14:20:29.320619Z info tf === Suite "telemetry_tracing_zipkin" run time: 2m50.769812322s === 2025-08-01T14:20:29.320639Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/trace.yaml 2025-08-01T14:20:29.320913Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:20:29.320917Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:20:29.491885Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/71513b04a3-8b77-4f56-91ad-abdc7f6ce12c 2025-08-01T14:20:29.492056Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1709s 2025-08-01T14:20:29.492906Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:29.582659Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/71b84202fa-1eaa-4d62-80a1-f3c250fc318b 2025-08-01T14:20:29.582887Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2617s 2025-08-01T14:20:29.583690Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:29.680781Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/71b4ed2c77-f1f2-46cf-ab4f-a08bf688a86a 2025-08-01T14:20:29.681008Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3598s 2025-08-01T14:20:29.681688Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:30.214219Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:20:30.214353Z info tf === SUCCEEDED: Cleanup Istio in 125.916µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin DONE 2 runs, 6 tests, 6 failures in 536.196s exec: [go test -json -test.run=^TestServerTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=true -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 4139008 2025-08-01T14:20:43.419048Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:20:43.419115Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-01T14:20:43.470823Z info tf === Test Framework Settings === 2025-08-01T14:20:43.470951Z info tf TestID: telemetry_tracing_zipkin RunID: aba3965d-32a2-47aa-91ef-23b3dc960a57 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-01T14:20:43.470961Z info tf =============================== 2025-08-01T14:20:43.471611Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-aba396 2025-08-01T14:20:43.471762Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-01T14:20:43.471785Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-01T14:20:43.471823Z info tf Using KubeConfigs: [/root/upi/cO/auth/kubeconfig]. 2025-08-01T14:20:43.471839Z info tf === BEGIN: Building clusters === 2025-08-01T14:20:43.477863Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cO/auth/kubeconfig 2025-08-01T14:20:43.477878Z info tf === DONE: Building clusters === 2025-08-01T14:20:43.477914Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-08-01T14:20:43.477958Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:20:43.477972Z info tf === Istio Component Config === 2025-08-01T14:20:43.478051Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-01T14:20:43.478063Z info tf ================================ 2025-08-01T14:20:43.482469Z info tf skipping deployment as specified in the config 2025-08-01T14:20:43.482522Z info tf === SUCCEEDED: Deploy Istio in 4.557458ms [Suite=telemetry_tracing_zipkin]=== 2025-08-01T14:20:43.482531Z info tf === BEGIN: Create namespace echo === 2025-08-01T14:20:43.506447Z info tf === SUCCEEDED: Create namespace echo in 23.907184ms === 2025-08-01T14:20:43.538152Z info tf === BEGIN: Deploy echo instances === 2025-08-01T14:20:43.656449Z info klog Waiting for caches to sync controller=echo 2025-08-01T14:20:43.656539Z info klog Waiting for caches to sync controller=echo 2025-08-01T14:20:43.757176Z info klog Caches are synced controller=echo 2025-08-01T14:20:43.757295Z info klog Caches are synced controller=echo 2025-08-01T14:21:52.485939Z info tf === SUCCEEDED: Deploy echo instances in 1m8.947781445s === 2025-08-01T14:21:52.532040Z info tf Checking pods ready... 2025-08-01T14:21:52.532078Z info tf Checking pods ready... 2025-08-01T14:21:52.543477Z info tf [ 0] zipkin-5c6fbc9d85-vkgsd Pending (Pending) 2025-08-01T14:21:52.744126Z info tf Checking pods ready... 2025-08-01T14:21:52.744161Z info tf Checking pods ready... 2025-08-01T14:21:52.747833Z info tf [ 0] zipkin-5c6fbc9d85-vkgsd Pending (Pending) 2025-08-01T14:21:53.148217Z info tf Checking pods ready... 2025-08-01T14:21:53.148259Z info tf Checking pods ready... 2025-08-01T14:21:53.152633Z info tf [ 0] zipkin-5c6fbc9d85-vkgsd Pending (Pending) 2025-08-01T14:21:53.953227Z info tf Checking pods ready... 2025-08-01T14:21:53.953306Z info tf Checking pods ready... 2025-08-01T14:21:53.957722Z info tf [ 0] zipkin-5c6fbc9d85-vkgsd Pending (Pending) 2025-08-01T14:21:55.558812Z info tf Checking pods ready... 2025-08-01T14:21:55.558858Z info tf Checking pods ready... 2025-08-01T14:21:55.563887Z info tf [ 0] zipkin-5c6fbc9d85-vkgsd Running (container not ready: 'zipkin') 2025-08-01T14:21:58.764046Z info tf Checking pods ready... 2025-08-01T14:21:58.764125Z info tf Checking pods ready... 2025-08-01T14:21:58.768058Z info tf [ 0] zipkin-5c6fbc9d85-vkgsd Running (container not ready: 'zipkin') 2025-08-01T14:22:01.968732Z info tf Checking pods ready... 2025-08-01T14:22:01.968791Z info tf Checking pods ready... 2025-08-01T14:22:01.972711Z info tf [ 0] zipkin-5c6fbc9d85-vkgsd Running (Ready) 2025-08-01T14:22:02.045168Z info tf === DONE: Setup: 'telemetry_tracing_zipkin' (1m18.567236211s) === 2025-08-01T14:22:02.045237Z info tf === BEGIN: Test Run: 'telemetry_tracing_zipkin' === === RUN TestServerTracing/primary-0 2025-08-01T14:22:02.045794Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:24:32.046256Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.000456985s)' === 2025-08-01T14:24:32.046398Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-08-01T14:24:32.046408Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-08-01T14:24:32.298076Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/787b4d2a9b-7903-4686-b938-c58cbfa37f71 2025-08-01T14:24:32.298211Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2514s 2025-08-01T14:24:32.299022Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:32.476690Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/783743b3a0-d52d-465d-b60d-6a57d7f74bea 2025-08-01T14:24:32.476937Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4301s 2025-08-01T14:24:32.478032Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:32.560567Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/783129f8c7-3ea0-4067-9634-231b905ad501 2025-08-01T14:24:32.560733Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5139s 2025-08-01T14:24:32.561528Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing/primary-0 (150.63s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing/primary-0 (re-run 2) (150.63s) === RUN TestServerTracing 2025-08-01T14:22:02.045645Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-08-01T14:24:32.679072Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.633418444s)' === 2025-08-01T14:24:32.679174Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-08-01T14:24:32.679207Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-08-01T14:24:32.868989Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/78ca37f7ba-550e-479f-9977-014464e8d72b 2025-08-01T14:24:32.869269Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1898s 2025-08-01T14:24:32.870053Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:32.953120Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/786a0a3afe-5901-4d4c-badd-90f7d521440e 2025-08-01T14:24:32.953281Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2738s 2025-08-01T14:24:32.954265Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:33.044692Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/783d38fb0d-488b-4ef4-b746-d3c45e7f678a 2025-08-01T14:24:33.044931Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3655s 2025-08-01T14:24:33.045578Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing (151.31s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing (re-run 2) (151.31s) 2025-08-01T14:24:33.359072Z info tf === FAILED: Test Run: 'telemetry_tracing_zipkin' (exitCode: 1) === 2025-08-01T14:24:33.359158Z info tf === Suite "telemetry_tracing_zipkin" run time: 3m49.881253663s === 2025-08-01T14:24:33.359181Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/trace.yaml 2025-08-01T14:24:33.359448Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:24:33.359447Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-08-01T14:24:33.599511Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/78aae4b3f2-ed1a-4b28-9d35-c3c97d4f48c2 2025-08-01T14:24:33.599638Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2400s 2025-08-01T14:24:33.600409Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:33.692362Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/783bf4418b-42ba-4826-a95a-df0ef85d9072 2025-08-01T14:24:33.692583Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3329s 2025-08-01T14:24:33.693179Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:33.791573Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/78cbab92b4-b68d-402a-a35a-d9bca7fa366d 2025-08-01T14:24:33.791786Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4320s 2025-08-01T14:24:33.792711Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:34.132223Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-08-01T14:24:34.132325Z info tf === SUCCEEDED: Cleanup Istio in 94.959µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin === Failed === FAIL: tests/integration/telemetry/tracing/zipkin TestClientTracing/primary-0 (150.74s) 2025-08-01T14:12:06.217350Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0]' === client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:49: Sending from cluster-0... client_tracing_test.go:43: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:14:36.218429Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0] (2m30.001064025s)' === 2025-08-01T14:14:36.218639Z error tf === Dumping Namespace echo State for TestClientTracing/primary-0... 2025-08-01T14:14:36.218638Z error tf === Dumping Istio Deployment State for TestClientTracing/primary-0... 2025-08-01T14:14:36.375563Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63537a53b2-a186-48b4-9152-25f5a4ae4005 2025-08-01T14:14:36.375700Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1567s 2025-08-01T14:14:36.376325Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:36.464301Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6363a59f29-53da-4f9b-8cae-a1043ab8aad4 2025-08-01T14:14:36.464456Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2454s 2025-08-01T14:14:36.465053Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:36.557039Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/637f8307b6-cd85-4d66-b630-a3f5a52e04b3 2025-08-01T14:14:36.557227Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3382s 2025-08-01T14:14:36.559184Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestClientTracing (151.46s) 2025-08-01T14:12:06.217208Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing]' === 2025-08-01T14:14:36.958810Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing] (2m30.741592037s)' === 2025-08-01T14:14:36.958952Z error tf === Dumping Namespace echo State for TestClientTracing... 2025-08-01T14:14:36.958951Z error tf === Dumping Istio Deployment State for TestClientTracing... 2025-08-01T14:14:37.335061Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/633f8a6ffa-135e-405f-abaa-636fa4de31ff 2025-08-01T14:14:37.335302Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3761s 2025-08-01T14:14:37.335889Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:37.438427Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63dc3008dc-6d67-4ded-b598-8b78ee00cebd 2025-08-01T14:14:37.438637Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4794s 2025-08-01T14:14:37.439464Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:14:37.538752Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63859214fc-f5b2-4d95-b410-dd2abd0c2725 2025-08-01T14:14:37.538976Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5797s 2025-08-01T14:14:37.539594Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing/primary-0 (151.12s) 2025-08-01T14:14:37.673729Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:17:07.675034Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.001293499s)' === 2025-08-01T14:17:07.675219Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-08-01T14:17:07.675218Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-08-01T14:17:08.038646Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/639c8f4cc5-6b7a-4c05-a8e3-876890b98e08 2025-08-01T14:17:08.038803Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3630s 2025-08-01T14:17:08.039690Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:08.191218Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63b3ffb3cb-54f7-4de5-8ec8-4fec42d17e86 2025-08-01T14:17:08.191390Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5155s 2025-08-01T14:17:08.192210Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:08.354197Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/63237c6e02-04db-46d1-8e33-e1f0803598ec 2025-08-01T14:17:08.354565Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6787s 2025-08-01T14:17:08.355108Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing (152.18s) 2025-08-01T14:14:37.673581Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-08-01T14:17:08.798270Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m31.124684067s)' === 2025-08-01T14:17:08.798388Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-08-01T14:17:08.798405Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-08-01T14:17:09.105817Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/630f050b10-c5f5-4b97-8286-f7418469aec3 2025-08-01T14:17:09.106180Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3075s 2025-08-01T14:17:09.106921Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:09.203665Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6377920fb9-194d-4b45-9403-5f6ff5023af3 2025-08-01T14:17:09.203775Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4051s 2025-08-01T14:17:09.205959Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:17:09.298355Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:12:06Z/6348eebaf8-0ee9-4f3d-87e6-104bb34fedb5 2025-08-01T14:17:09.298609Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4999s 2025-08-01T14:17:09.300770Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing/primary-0 (re-run 1) (150.64s) 2025-08-01T14:17:57.915132Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:20:27.916433Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.001290686s)' === 2025-08-01T14:20:27.916621Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-08-01T14:20:27.916621Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-08-01T14:20:28.043429Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/715a9dc70c-f3d9-4f76-801d-58bac01acfe4 2025-08-01T14:20:28.043670Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1267s 2025-08-01T14:20:28.044252Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:28.137613Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/71e5b33997-5f4c-41e7-98b8-58e7e027278d 2025-08-01T14:20:28.137700Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2207s 2025-08-01T14:20:28.138323Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:28.220865Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/718b810261-a217-431f-969e-3b4b0dab22bf 2025-08-01T14:20:28.221085Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3041s 2025-08-01T14:20:28.221648Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing (re-run 1) (151.41s) 2025-08-01T14:17:57.914917Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-08-01T14:20:28.551574Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.63664712s)' === 2025-08-01T14:20:28.551705Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-08-01T14:20:28.551721Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-08-01T14:20:28.839533Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/714cd265d9-4c04-4a58-ac01-0aa277fa7578 2025-08-01T14:20:28.839749Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2877s 2025-08-01T14:20:28.841615Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:28.998680Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/71fc40688d-3932-4c2a-8e2f-dabf2708df3d 2025-08-01T14:20:28.998950Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4469s 2025-08-01T14:20:28.999510Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:20:29.072220Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:17:58Z/7164d090a2-0191-40dd-b1fc-45e712071e7b 2025-08-01T14:20:29.072372Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5203s 2025-08-01T14:20:29.073112Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing/primary-0 (re-run 2) (150.63s) 2025-08-01T14:22:02.045794Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-08-01T14:24:32.046256Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.000456985s)' === 2025-08-01T14:24:32.046398Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-08-01T14:24:32.046408Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-08-01T14:24:32.298076Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/787b4d2a9b-7903-4686-b938-c58cbfa37f71 2025-08-01T14:24:32.298211Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2514s 2025-08-01T14:24:32.299022Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:32.476690Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/783743b3a0-d52d-465d-b60d-6a57d7f74bea 2025-08-01T14:24:32.476937Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4301s 2025-08-01T14:24:32.478032Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:32.560567Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/783129f8c7-3ea0-4067-9634-231b905ad501 2025-08-01T14:24:32.560733Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5139s 2025-08-01T14:24:32.561528Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing (re-run 2) (151.31s) 2025-08-01T14:22:02.045645Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-08-01T14:24:32.679072Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.633418444s)' === 2025-08-01T14:24:32.679174Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-08-01T14:24:32.679207Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-08-01T14:24:32.868989Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/78ca37f7ba-550e-479f-9977-014464e8d72b 2025-08-01T14:24:32.869269Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1898s 2025-08-01T14:24:32.870053Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:32.953120Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/786a0a3afe-5901-4d4c-badd-90f7d521440e 2025-08-01T14:24:32.953281Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2738s 2025-08-01T14:24:32.954265Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-01T14:24:33.044692Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-01T14:22:02Z/783d38fb0d-488b-4ef4-b746-d3c45e7f678a 2025-08-01T14:24:33.044931Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3655s 2025-08-01T14:24:33.045578Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 3 runs, 8 tests, 8 failures in 780.114s exec: go version