exec: [go test -json -tags=integ -timeout 180m ./tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3531156 2025-06-24T15:19:11.349765Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:19:11.349810Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:19:11.379845Z info tf === Test Framework Settings === 2025-06-24T15:19:11.379908Z info tf TestID: telemetry_tracing_zipkin RunID: 94684e6e-d916-424e-b357-3b194fb88efb NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:19:11.379914Z info tf =============================== 2025-06-24T15:19:11.381164Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-94684e 2025-06-24T15:19:11.381236Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:19:11.381247Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:19:11.381262Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:19:11.381269Z info tf === BEGIN: Building clusters === 2025-06-24T15:19:11.383579Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:19:11.383587Z info tf === DONE: Building clusters === 2025-06-24T15:19:11.383602Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-06-24T15:19:11.383622Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T15:19:11.383627Z info tf === Istio Component Config === 2025-06-24T15:19:11.383661Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:19:11.383665Z info tf ================================ 2025-06-24T15:19:11.386163Z info tf skipping deployment as specified in the config 2025-06-24T15:19:11.386176Z info tf === SUCCEEDED: Deploy Istio in 2.551382ms [Suite=telemetry_tracing_zipkin]=== 2025-06-24T15:19:11.386183Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:19:11.437396Z info tf === SUCCEEDED: Create namespace echo in 51.203475ms === 2025-06-24T15:19:11.500662Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:19:11.723010Z info klog Waiting for caches to sync for echo 2025-06-24T15:19:11.725881Z info klog Waiting for caches to sync for echo 2025-06-24T15:19:11.824177Z info klog Caches are synced for echo 2025-06-24T15:19:11.826858Z info klog Caches are synced for echo 2025-06-24T15:19:21.332941Z info tf === SUCCEEDED: Deploy echo instances in 9.83227158s === 2025-06-24T15:19:21.462359Z info tf Checking pods ready... 2025-06-24T15:19:21.462389Z info tf Checking pods ready... 2025-06-24T15:19:21.497625Z info tf [ 0] zipkin-78496844d5-9wnb9 Pending (Pending) 2025-06-24T15:19:21.698368Z info tf Checking pods ready... 2025-06-24T15:19:21.698428Z info tf Checking pods ready... 2025-06-24T15:19:21.703345Z info tf [ 0] zipkin-78496844d5-9wnb9 Pending (Pending) 2025-06-24T15:19:22.104126Z info tf Checking pods ready... 2025-06-24T15:19:22.104160Z info tf Checking pods ready... 2025-06-24T15:19:22.109324Z info tf [ 0] zipkin-78496844d5-9wnb9 Pending (Pending) 2025-06-24T15:19:22.910004Z info tf Checking pods ready... 2025-06-24T15:19:22.910037Z info tf Checking pods ready... 2025-06-24T15:19:22.915187Z info tf [ 0] zipkin-78496844d5-9wnb9 Pending (Pending) 2025-06-24T15:19:24.515281Z info tf Checking pods ready... 2025-06-24T15:19:24.515318Z info tf Checking pods ready... 2025-06-24T15:19:24.522453Z info tf [ 0] zipkin-78496844d5-9wnb9 Pending (Pending) 2025-06-24T15:19:27.723458Z info tf Checking pods ready... 2025-06-24T15:19:27.723556Z info tf Checking pods ready... 2025-06-24T15:19:27.730330Z info tf [ 0] zipkin-78496844d5-9wnb9 Pending (Pending) 2025-06-24T15:19:30.930718Z info tf Checking pods ready... 2025-06-24T15:19:30.930757Z info tf Checking pods ready... 2025-06-24T15:19:30.936221Z info tf [ 0] zipkin-78496844d5-9wnb9 Pending (Pending) 2025-06-24T15:19:34.136299Z info tf Checking pods ready... 2025-06-24T15:19:34.136347Z info tf Checking pods ready... 2025-06-24T15:19:34.143863Z info tf [ 0] zipkin-78496844d5-9wnb9 Running (container not ready: 'zipkin') 2025-06-24T15:19:37.344259Z info tf Checking pods ready... 2025-06-24T15:19:37.344307Z info tf Checking pods ready... 2025-06-24T15:19:37.350484Z info tf [ 0] zipkin-78496844d5-9wnb9 Running (container not ready: 'zipkin') 2025-06-24T15:19:40.551369Z info tf Checking pods ready... 2025-06-24T15:19:40.551419Z info tf Checking pods ready... 2025-06-24T15:19:40.555952Z info tf [ 0] zipkin-78496844d5-9wnb9 Running (container not ready: 'zipkin') 2025-06-24T15:19:43.756185Z info tf Checking pods ready... 2025-06-24T15:19:43.756226Z info tf Checking pods ready... 2025-06-24T15:19:43.761159Z info tf [ 0] zipkin-78496844d5-9wnb9 Running (Ready) 2025-06-24T15:19:43.838762Z info tf === DONE: Setup: 'telemetry_tracing_zipkin' (32.4551485s) === 2025-06-24T15:19:43.838801Z info tf === BEGIN: Test Run: 'telemetry_tracing_zipkin' === === RUN TestClientTracing/primary-0 2025-06-24T15:19:43.839582Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0]' === client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... 2025-06-24T15:20:25.385758Z error port forward failed: lost connection to pod 2025-06-24T15:20:25.825583Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward terminated 2025-06-24T15:20:25.860180Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward reconnect success client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... 2025-06-24T15:20:43.790723Z error port forward failed: lost connection to pod client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... 2025-06-24T15:21:31.865048Z error port forward failed: lost connection to pod 2025-06-24T15:21:32.325677Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward terminated 2025-06-24T15:21:32.362125Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward reconnect success client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:44: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:22:13.839905Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0] (2m30.000322449s)' === 2025-06-24T15:22:13.839968Z error tf === Dumping Namespace echo State for TestClientTracing/primary-0... 2025-06-24T15:22:13.839961Z error tf === Dumping Istio Deployment State for TestClientTracing/primary-0... 2025-06-24T15:22:14.130086Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/7086ddec6-d51a-4a47-893b-bcb48be642fd 2025-06-24T15:22:14.130387Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2901s 2025-06-24T15:22:14.131250Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:22:14.232660Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/715977707-6486-4a68-92ef-25a4eabd3321 2025-06-24T15:22:14.232840Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3925s 2025-06-24T15:22:14.233512Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:22:14.337093Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/7a87d58dd-4681-4cfc-9922-ba9b1aac1891 2025-06-24T15:22:14.337306Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4969s 2025-06-24T15:22:14.338016Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestClientTracing/primary-0 (150.91s) FAIL tests/integration/telemetry/tracing/zipkin.TestClientTracing/primary-0 (150.91s) === RUN TestClientTracing 2025-06-24T15:19:43.839291Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing]' === 2025-06-24T15:22:14.752615Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing] (2m30.913318457s)' === 2025-06-24T15:22:14.752777Z error tf === Dumping Namespace echo State for TestClientTracing... 2025-06-24T15:22:14.752772Z error tf === Dumping Istio Deployment State for TestClientTracing... 2025-06-24T15:22:14.891039Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/770d07cf4-32f3-4e3f-96b1-a78d350f8f99 2025-06-24T15:22:14.891322Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1383s 2025-06-24T15:22:14.891735Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:22:14.994916Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/7a59dab1a-ae50-41c6-bd8b-3b4f190319f3 2025-06-24T15:22:14.995002Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2420s 2025-06-24T15:22:14.995764Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:22:15.110992Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/7e1fe8bc4-59fa-41bf-917f-ede483841f87 2025-06-24T15:22:15.111160Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3581s 2025-06-24T15:22:15.111714Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestClientTracing (151.70s) FAIL tests/integration/telemetry/tracing/zipkin.TestClientTracing (151.70s) === RUN TestServerTracing/primary-0 2025-06-24T15:22:15.543518Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T15:22:38.366909Z error port forward failed: lost connection to pod 2025-06-24T15:22:38.825591Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward terminated 2025-06-24T15:22:38.866288Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T15:23:44.871413Z error port forward failed: lost connection to pod 2025-06-24T15:23:45.325128Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward terminated 2025-06-24T15:23:45.363642Z warn pod: echo/server-v1-6f45fcb457-cnbfh port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:24:45.544494Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.000972189s)' === 2025-06-24T15:24:45.544623Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-06-24T15:24:45.544637Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-06-24T15:24:46.703814Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/7f153c8c0-66a9-4d6c-b51e-3a462441623c 2025-06-24T15:24:46.704114Z info tf istioctl ([x internal-debug --all configz]): completed after 1.1592s 2025-06-24T15:24:46.704680Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:24:46.886927Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/74e800c3f-fafa-417a-b8d1-8506947e6510 2025-06-24T15:24:46.887084Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.3421s 2025-06-24T15:24:46.887706Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:24:46.980417Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/702f6c981-1cc5-4927-b3cb-53441d50e1e3 2025-06-24T15:24:46.980598Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.4356s 2025-06-24T15:24:46.981307Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing/primary-0 (151.53s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing/primary-0 (151.53s) === RUN TestServerTracing 2025-06-24T15:22:15.543309Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-06-24T15:24:47.072882Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m31.529569833s)' === 2025-06-24T15:24:47.073008Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-06-24T15:24:47.073004Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-06-24T15:24:47.243580Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/77e8834c4-9395-49c6-873d-9277ddeeea9b 2025-06-24T15:24:47.243834Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1706s 2025-06-24T15:24:47.244234Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:24:47.382814Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/72a17852f-3567-4c1b-b601-9fed9719f7eb 2025-06-24T15:24:47.382968Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3097s 2025-06-24T15:24:47.383555Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:24:47.503083Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/74f7d05ec-f5de-4085-b17c-2ff9b3542d5d 2025-06-24T15:24:47.503254Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4300s 2025-06-24T15:24:47.504041Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing (152.31s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing (152.31s) 2025-06-24T15:24:47.852222Z info tf === FAILED: Test Run: 'telemetry_tracing_zipkin' (exitCode: 1) === 2025-06-24T15:24:47.852259Z info tf === Suite "telemetry_tracing_zipkin" run time: 5m36.468662917s === 2025-06-24T15:24:47.852270Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/trace.yaml 2025-06-24T15:24:47.853029Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:24:47.853048Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:24:48.052989Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/741b223fd-e8d2-4bfc-a4db-2a2a259bcda2 2025-06-24T15:24:48.053232Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2000s 2025-06-24T15:24:48.053926Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:24:48.157488Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/7ed837e80-89e5-475b-87bd-3996ec3275c7 2025-06-24T15:24:48.157585Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3044s 2025-06-24T15:24:48.158126Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:24:48.259895Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:19:43Z/781d465ac-07af-477c-b505-83775e9d823a 2025-06-24T15:24:48.260094Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4067s 2025-06-24T15:24:48.260698Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:24:48.769200Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T15:24:48.769307Z info tf === SUCCEEDED: Cleanup Istio in 94.116µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin DONE 4 tests, 4 failures in 350.485s exec: [go test -json -test.run=^TestClientTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3533221 2025-06-24T15:24:59.954194Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:24:59.954237Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:24:59.983573Z info tf === Test Framework Settings === 2025-06-24T15:24:59.983688Z info tf TestID: telemetry_tracing_zipkin RunID: 3b19bfcf-1650-4619-b451-cff8df037978 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:24:59.983697Z info tf =============================== 2025-06-24T15:24:59.983824Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-3b19bf 2025-06-24T15:24:59.983894Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:24:59.983903Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:24:59.983920Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:24:59.983927Z info tf === BEGIN: Building clusters === 2025-06-24T15:24:59.986393Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:24:59.986401Z info tf === DONE: Building clusters === 2025-06-24T15:24:59.986415Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-06-24T15:24:59.986436Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T15:24:59.986440Z info tf === Istio Component Config === 2025-06-24T15:24:59.986468Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:24:59.986472Z info tf ================================ 2025-06-24T15:24:59.989138Z info tf skipping deployment as specified in the config 2025-06-24T15:24:59.989154Z info tf === SUCCEEDED: Deploy Istio in 2.714321ms [Suite=telemetry_tracing_zipkin]=== 2025-06-24T15:24:59.989163Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:25:00.007067Z info tf === SUCCEEDED: Create namespace echo in 17.889722ms === 2025-06-24T15:25:00.042294Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:25:00.048344Z error tf === FAILED: Deploy echo instances === 2025-06-24T15:25:00.048367Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-3b19bf/_suite_context/Service.3368423744.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:25:00.048394Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:25:00.048422Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:25:00.137262Z error klog an error occurred forwarding 33299 -> 15020: error forwarding port 15020 to pod 96b59f1145f424b2c89bb386bba7d4795f6dab9609963782e1bffd0c74b80bb3, uid : port forward into network namespace "/var/run/netns/5a29143f-9a87-438f-a2f3-6d52a66b0c08": failed to connect to localhost:15020 inside namespace 96b59f1145f424b2c89bb386bba7d4795f6dab9609963782e1bffd0c74b80bb3: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:25:00.139319Z error port forward failed: lost connection to pod 2025-06-24T15:25:00.139319Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-thkzh for: Get "http://localhost:33299/debug/ndsz": EOF 2025-06-24T15:25:00.142449Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-thkzh for: Get "http://localhost:33299/stats/prometheus": dial tcp 127.0.0.1:33299: connect: connection refused 2025-06-24T15:25:00.166926Z error klog an error occurred forwarding 44009 -> 15020: error forwarding port 15020 to pod 2c4d42539b37343980de94227bb177df297a774fbd6a2f79006c6ffdb836181d, uid : port forward into network namespace "/var/run/netns/640c1249-81df-48f2-a067-afc581f1a4bb": failed to connect to localhost:15020 inside namespace 2c4d42539b37343980de94227bb177df297a774fbd6a2f79006c6ffdb836181d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:25:00.168496Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-cnbfh for: Get "http://localhost:44009/debug/ndsz": EOF 2025-06-24T15:25:00.168476Z error port forward failed: lost connection to pod 2025-06-24T15:25:00.169391Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-cnbfh for: Get "http://localhost:44009/stats/prometheus": dial tcp 127.0.0.1:44009: connect: connection refused 2025-06-24T15:25:00.175226Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:24:54Z/992c89db3-3599-4f4b-9b1a-73df7c4c2b9f 2025-06-24T15:25:00.175347Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.1267s 2025-06-24T15:25:00.175950Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:25:00.334460Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:24:54Z/9048d3c57-5dd2-44f7-9029-536ddf389ba4 2025-06-24T15:25:00.334635Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2859s 2025-06-24T15:25:00.335396Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:25:00.465635Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:24:54Z/962a03e51-480c-40fc-b613-4700813b7d11 2025-06-24T15:25:00.465764Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4171s 2025-06-24T15:25:00.466343Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:25:01.449790Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-thkzh for: Get "http://localhost:40135/config_dump?include_eds=true": EOF 2025-06-24T15:25:01.455210Z error klog an error occurred forwarding 40135 -> 15000: error forwarding port 15000 to pod 96b59f1145f424b2c89bb386bba7d4795f6dab9609963782e1bffd0c74b80bb3, uid : sandbox 96b59f1145f424b2c89bb386bba7d4795f6dab9609963782e1bffd0c74b80bb3 is not running: Unhandled Error logger=UnhandledError 2025-06-24T15:25:01.457003Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-thkzh for: Get "http://localhost:40135/clusters": EOF 2025-06-24T15:25:01.457013Z error port forward failed: lost connection to pod 2025-06-24T15:25:01.465377Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-cnbfh for: Get "http://localhost:39037/config_dump?include_eds=true": EOF 2025-06-24T15:25:01.471216Z error klog an error occurred forwarding 39037 -> 15000: error forwarding port 15000 to pod 2c4d42539b37343980de94227bb177df297a774fbd6a2f79006c6ffdb836181d, uid : port forward into network namespace "/var/run/netns/640c1249-81df-48f2-a067-afc581f1a4bb": failed to connect to localhost:15000 inside namespace 2c4d42539b37343980de94227bb177df297a774fbd6a2f79006c6ffdb836181d: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:25:01.473194Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-cnbfh for: Get "http://localhost:39037/clusters": EOF 2025-06-24T15:25:01.473239Z error port forward failed: lost connection to pod 2025-06-24T15:25:01.477475Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-3b19bf/_suite_context/Service.3368423744.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:25:01.477501Z info tf === FAILED: Setup: 'telemetry_tracing_zipkin' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-3b19bf/_suite_context/Service.3368423744.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T15:25:01.477511Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-3b19bf/_suite_context/Service.3368423744.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:25:01.477546Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:25:01.477546Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:25:01.591340Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-cnbfh_echo") 2025-06-24T15:25:01.610840Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-5c7cd449cf-thkzh_echo") 2025-06-24T15:25:01.614007Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-5c7cd449cf-thkzh_echo") 2025-06-24T15:25:01.620938Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-cnbfh_echo") 2025-06-24T15:25:01.637603Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:24:54Z/9dbce6158-8b96-4a63-ac0e-25eb4a31dab7 2025-06-24T15:25:01.637716Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1599s 2025-06-24T15:25:01.638324Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:25:01.654109Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-cnbfh_echo") 2025-06-24T15:25:01.693947Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-cnbfh_echo") 2025-06-24T15:25:01.724135Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:25:01.736639Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:24:54Z/979ca8b27-c5f1-4a2c-84eb-beb9054f1365 2025-06-24T15:25:01.736749Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2589s 2025-06-24T15:25:01.737445Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:25:01.749798Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:25:01.821946Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:24:54Z/90e06cb68-a088-472d-96ed-1d860ac86abe 2025-06-24T15:25:01.822131Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3443s 2025-06-24T15:25:01.822693Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:25:02.129516Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-cnbfh/app for: pods "server-v1-6f45fcb457-cnbfh" not found 2025-06-24T15:25:02.132367Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-cnbfh/istio-proxy for: pods "server-v1-6f45fcb457-cnbfh" not found 2025-06-24T15:25:02.135144Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-cnbfh/istio-validation for: pods "server-v1-6f45fcb457-cnbfh" not found 2025-06-24T15:25:02.138824Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T15:25:02.138849Z info tf === SUCCEEDED: Cleanup Istio in 17.451µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin exec: [go test -json -test.run=^TestServerTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3534451 2025-06-24T15:25:13.400962Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:25:13.401024Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:25:13.435413Z info tf === Test Framework Settings === 2025-06-24T15:25:13.435514Z info tf TestID: telemetry_tracing_zipkin RunID: e99fe762-f9aa-4300-b3a8-38e0669e5e57 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:25:13.435519Z info tf =============================== 2025-06-24T15:25:13.435644Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-e99fe7 2025-06-24T15:25:13.435714Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:25:13.435725Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:25:13.435740Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:25:13.435746Z info tf === BEGIN: Building clusters === 2025-06-24T15:25:13.438127Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:25:13.438135Z info tf === DONE: Building clusters === 2025-06-24T15:25:13.438148Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-06-24T15:25:13.438170Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T15:25:13.438175Z info tf === Istio Component Config === 2025-06-24T15:25:13.438203Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:25:13.438207Z info tf ================================ 2025-06-24T15:25:13.440784Z info tf skipping deployment as specified in the config 2025-06-24T15:25:13.440798Z info tf === SUCCEEDED: Deploy Istio in 2.625192ms [Suite=telemetry_tracing_zipkin]=== 2025-06-24T15:25:13.440804Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:25:13.490666Z info tf === SUCCEEDED: Create namespace echo in 49.850658ms === 2025-06-24T15:25:13.568717Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:25:13.878835Z info klog Waiting for caches to sync for echo 2025-06-24T15:25:13.879589Z info klog Waiting for caches to sync for echo 2025-06-24T15:25:13.979418Z info klog Caches are synced for echo 2025-06-24T15:25:13.979826Z info klog Caches are synced for echo 2025-06-24T15:25:23.486453Z info tf === SUCCEEDED: Deploy echo instances in 9.917729374s === 2025-06-24T15:25:23.584120Z info tf Checking pods ready... 2025-06-24T15:25:23.584206Z info tf Checking pods ready... 2025-06-24T15:25:23.606434Z info tf [ 0] zipkin-78496844d5-v7kdw Pending (Pending) 2025-06-24T15:25:23.807247Z info tf Checking pods ready... 2025-06-24T15:25:23.807289Z info tf Checking pods ready... 2025-06-24T15:25:23.811767Z info tf [ 0] zipkin-78496844d5-v7kdw Pending (Pending) 2025-06-24T15:25:24.211895Z info tf Checking pods ready... 2025-06-24T15:25:24.211924Z info tf Checking pods ready... 2025-06-24T15:25:24.217290Z info tf [ 0] zipkin-78496844d5-v7kdw Pending (Pending) 2025-06-24T15:25:25.017530Z info tf Checking pods ready... 2025-06-24T15:25:25.017574Z info tf Checking pods ready... 2025-06-24T15:25:25.023106Z info tf [ 0] zipkin-78496844d5-v7kdw Pending (Pending) 2025-06-24T15:25:26.623292Z info tf Checking pods ready... 2025-06-24T15:25:26.623323Z info tf Checking pods ready... 2025-06-24T15:25:26.629547Z info tf [ 0] zipkin-78496844d5-v7kdw Pending (Pending) 2025-06-24T15:25:29.829836Z info tf Checking pods ready... 2025-06-24T15:25:29.829889Z info tf Checking pods ready... 2025-06-24T15:25:29.835002Z info tf [ 0] zipkin-78496844d5-v7kdw Pending (Pending) 2025-06-24T15:25:33.035887Z info tf Checking pods ready... 2025-06-24T15:25:33.035941Z info tf Checking pods ready... 2025-06-24T15:25:33.042182Z info tf [ 0] zipkin-78496844d5-v7kdw Pending (Pending) 2025-06-24T15:25:36.243074Z info tf Checking pods ready... 2025-06-24T15:25:36.243129Z info tf Checking pods ready... 2025-06-24T15:25:36.251100Z info tf [ 0] zipkin-78496844d5-v7kdw Running (container not ready: 'zipkin') 2025-06-24T15:25:39.452201Z info tf Checking pods ready... 2025-06-24T15:25:39.452232Z info tf Checking pods ready... 2025-06-24T15:25:39.456849Z info tf [ 0] zipkin-78496844d5-v7kdw Running (Ready) 2025-06-24T15:25:39.521485Z info tf === DONE: Setup: 'telemetry_tracing_zipkin' (26.083328759s) === 2025-06-24T15:25:39.521506Z info tf === BEGIN: Test Run: 'telemetry_tracing_zipkin' === === RUN TestServerTracing/primary-0 2025-06-24T15:25:39.521958Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T15:26:28.350195Z error port forward failed: lost connection to pod 2025-06-24T15:26:28.522272Z warn pod: echo/server-v1-6f45fcb457-zbwtr port forward terminated 2025-06-24T15:26:28.576805Z warn pod: echo/server-v1-6f45fcb457-zbwtr port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T15:26:39.484411Z error port forward failed: lost connection to pod server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T15:27:34.583152Z error port forward failed: lost connection to pod 2025-06-24T15:27:35.022345Z warn pod: echo/server-v1-6f45fcb457-zbwtr port forward terminated 2025-06-24T15:27:35.082606Z warn pod: echo/server-v1-6f45fcb457-zbwtr port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:28:09.522364Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.000399855s)' === 2025-06-24T15:28:09.522446Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-06-24T15:28:09.522437Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-06-24T15:28:09.727270Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/15ffd95c8e-a33a-46ee-be99-090eb021ae28 2025-06-24T15:28:09.727716Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2049s 2025-06-24T15:28:09.728026Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:28:09.807434Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/157e9c9ca0-d798-433a-bb60-914e95d9775b 2025-06-24T15:28:09.807644Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2848s 2025-06-24T15:28:09.808467Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:28:09.880737Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/1530380659-7cf1-497d-a6fb-b2b4d116ff0b 2025-06-24T15:28:09.880852Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3580s 2025-06-24T15:28:09.881358Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing/primary-0 (150.78s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing/primary-0 (re-run 1) (150.78s) === RUN TestServerTracing 2025-06-24T15:25:39.521814Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-06-24T15:28:10.302422Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.780604492s)' === 2025-06-24T15:28:10.302492Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-06-24T15:28:10.302502Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-06-24T15:28:10.424210Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/15dc42ee47-111f-4606-a78f-20e11e936501 2025-06-24T15:28:10.424486Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1218s 2025-06-24T15:28:10.425043Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:28:10.502429Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/15efaee99a-72ab-43ca-b82d-b1358efaf86d 2025-06-24T15:28:10.502610Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1999s 2025-06-24T15:28:10.503284Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:28:10.572771Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/158042d289-afe6-4263-a34a-c7ae88be22ad 2025-06-24T15:28:10.572932Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2702s 2025-06-24T15:28:10.573636Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing (151.52s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing (re-run 1) (151.52s) 2025-06-24T15:28:11.046636Z info tf === FAILED: Test Run: 'telemetry_tracing_zipkin' (exitCode: 1) === 2025-06-24T15:28:11.046705Z info tf === Suite "telemetry_tracing_zipkin" run time: 2m57.608554669s === 2025-06-24T15:28:11.046728Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/trace.yaml 2025-06-24T15:28:11.046962Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:28:11.046962Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T15:28:11.157029Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/154f3fcd36-d0cf-492b-b8f9-6fe053bb35f0 2025-06-24T15:28:11.157274Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1101s 2025-06-24T15:28:11.157827Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:28:11.236573Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/15d131f64d-25ce-460d-b51a-9514764fded9 2025-06-24T15:28:11.236717Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1895s 2025-06-24T15:28:11.237443Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:28:11.315665Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:25:39Z/15b218af04-dfc9-430a-aed9-da10a8358e3f 2025-06-24T15:28:11.315985Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2687s 2025-06-24T15:28:11.316672Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:28:11.906394Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T15:28:11.906467Z info tf === SUCCEEDED: Cleanup Istio in 59.654µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin DONE 2 runs, 6 tests, 6 failures in 553.602s exec: [go test -json -test.run=^TestServerTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3535914