PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:51:21.612899Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.312958885s)' === 2025-06-24T10:51:21.612982Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:51:21.613019Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:51:21.769858Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67ae3db5be-f33e-488c-af41-1e7dfe73d8cd 2025-06-24T10:51:21.770240Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1570s 2025-06-24T10:51:21.770550Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:21.898879Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/673878c157-0b0c-41e6-9397-14fc13415622 2025-06-24T10:51:21.899069Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2858s 2025-06-24T10:51:21.899849Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:22.009235Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67407c836d-b4b4-435b-b20f-4d0b304dc923 2025-06-24T10:51:22.009392Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3961s 2025-06-24T10:51:22.009981Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 3 runs, 24 tests, 24 failures in 1716.047s exec: go version [root@cpt-ppc-059 istio]# login as: root Authenticating with public key "swapnali@abc.com" Register this system with Red Hat Insights: insights-client --register Create an account or view all your systems at https://red.ht/insights-dashboard Last login: Tue Jun 24 05:42:49 2025 from 10.74.80.163 [root@cpt-ppc-059 ~]# clear [root@cpt-ppc-059 ~]# export HUB=quay.io/maistra [root@cpt-ppc-059 ~]# export TAG=ibm-p [root@cpt-ppc-059 ~]# export KUBECONFIG=/root/upi/cM/auth/kubeconfig [root@cpt-ppc-059 ~]# oc login https://api.ppc64le-qe59m.ibmp.psi.redhat.com:6443 -u jenkins -p Jenkins-PSI-01-ibmpass --insecure-skip-tls-verify WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 72 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". [root@cpt-ppc-059 ~]# cd swapnali/check_3.0_tp2/ [root@cpt-ppc-059 check_3.0_tp2]# ls check_collec GA istio sail-operator [root@cpt-ppc-059 check_3.0_tp2]# cd istio/ [root@cpt-ppc-059 istio]# gotestsum -f testname --junitfile-project-name istio --junitfile /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/junit-telemetry-tracing-zipkin.xml --rerun-fails --packages=./tests/integration/telemetry/tracing/zipkin --rerun-fails-max-failures=30 --debug -- -tags=integ -timeout 180m -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts exec: [go test -json -tags=integ -timeout 180m ./tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3468652 2025-06-24T11:47:11.527243Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:47:11.527299Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:47:11.561502Z info tf === Test Framework Settings === 2025-06-24T11:47:11.561622Z info tf TestID: telemetry_tracing_zipkin RunID: e4c9d277-c19d-44cc-9f43-cebb3d0847f9 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T11:47:11.561628Z info tf =============================== 2025-06-24T11:47:11.616678Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-e4c9d2 2025-06-24T11:47:11.616847Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T11:47:11.616865Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T11:47:11.616894Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T11:47:11.616910Z info tf === BEGIN: Building clusters === 2025-06-24T11:47:11.619356Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T11:47:11.619363Z info tf === DONE: Building clusters === 2025-06-24T11:47:11.619395Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-06-24T11:47:11.625897Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:47:11.625907Z info tf === Istio Component Config === 2025-06-24T11:47:11.625945Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T11:47:11.625952Z info tf ================================ 2025-06-24T11:47:11.628665Z info tf skipping deployment as specified in the config 2025-06-24T11:47:11.628682Z info tf === SUCCEEDED: Deploy Istio in 2.779309ms [Suite=telemetry_tracing_zipkin]=== 2025-06-24T11:47:11.628690Z info tf === BEGIN: Create namespace echo === 2025-06-24T11:47:11.653781Z info tf === SUCCEEDED: Create namespace echo in 25.079785ms === 2025-06-24T11:47:11.724519Z info tf === BEGIN: Deploy echo instances === 2025-06-24T11:47:11.877557Z info klog Waiting for caches to sync for echo 2025-06-24T11:47:11.909006Z info klog Waiting for caches to sync for echo 2025-06-24T11:47:11.977974Z info klog Caches are synced for echo 2025-06-24T11:47:12.009637Z info klog Caches are synced for echo 2025-06-24T11:47:18.314935Z info tf === SUCCEEDED: Deploy echo instances in 6.590426209s === 2025-06-24T11:47:18.368891Z info tf Checking pods ready... 2025-06-24T11:47:18.368966Z info tf Checking pods ready... 2025-06-24T11:47:18.375496Z info tf [ 0] zipkin-77cbd75c85-nw2fb Pending (Pending) 2025-06-24T11:47:18.575562Z info tf Checking pods ready... 2025-06-24T11:47:18.575671Z info tf Checking pods ready... 2025-06-24T11:47:18.581290Z info tf [ 0] zipkin-77cbd75c85-nw2fb Pending (Pending) 2025-06-24T11:47:18.982365Z info tf Checking pods ready... 2025-06-24T11:47:18.982476Z info tf Checking pods ready... 2025-06-24T11:47:18.991291Z info tf [ 0] zipkin-77cbd75c85-nw2fb Pending (Pending) 2025-06-24T11:47:19.791649Z info tf Checking pods ready... 2025-06-24T11:47:19.791776Z info tf Checking pods ready... 2025-06-24T11:47:19.795848Z info tf [ 0] zipkin-77cbd75c85-nw2fb Pending (Pending) 2025-06-24T11:47:21.396869Z info tf Checking pods ready... 2025-06-24T11:47:21.396955Z info tf Checking pods ready... 2025-06-24T11:47:21.402630Z info tf [ 0] zipkin-77cbd75c85-nw2fb Running (container not ready: 'zipkin') 2025-06-24T11:47:24.603689Z info tf Checking pods ready... 2025-06-24T11:47:24.603771Z info tf Checking pods ready... 2025-06-24T11:47:24.608596Z info tf [ 0] zipkin-77cbd75c85-nw2fb Running (Ready) 2025-06-24T11:47:24.675087Z info tf === DONE: Setup: 'telemetry_tracing_zipkin' (13.055679175s) === 2025-06-24T11:47:24.675144Z info tf === BEGIN: Test Run: 'telemetry_tracing_zipkin' === === RUN TestClientTracing/primary-0 2025-06-24T11:47:24.675741Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0]' === client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... 2025-06-24T11:48:24.245020Z error port forward failed: lost connection to pod 2025-06-24T11:48:24.512970Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:48:24.549163Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success 2025-06-24T11:48:24.637345Z error port forward failed: lost connection to pod client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... 2025-06-24T11:49:30.552698Z error port forward failed: lost connection to pod 2025-06-24T11:49:31.013657Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:49:31.046295Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:44: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T11:49:54.677083Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0] (2m30.001336075s)' === 2025-06-24T11:49:54.677222Z error tf === Dumping Istio Deployment State for TestClientTracing/primary-0... 2025-06-24T11:49:54.677241Z error tf === Dumping Namespace echo State for TestClientTracing/primary-0... 2025-06-24T11:49:54.906055Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8b41ce590-d236-463a-bbf0-9ee8ea7235a3 2025-06-24T11:49:54.906321Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2288s 2025-06-24T11:49:54.906808Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:54.988377Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8893696be-8faa-4c67-bf5f-622a94e77b61 2025-06-24T11:49:54.988482Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3109s 2025-06-24T11:49:54.989120Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:55.081790Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/88a60e234-41a2-4ee2-b1ca-61343613d67a 2025-06-24T11:49:55.081964Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4043s 2025-06-24T11:49:55.082555Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestClientTracing/primary-0 (150.58s) FAIL tests/integration/telemetry/tracing/zipkin.TestClientTracing/primary-0 (150.58s) === RUN TestClientTracing 2025-06-24T11:47:24.675555Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing]' === 2025-06-24T11:49:55.251281Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing] (2m30.575722903s)' === 2025-06-24T11:49:55.251363Z error tf === Dumping Namespace echo State for TestClientTracing... 2025-06-24T11:49:55.251367Z error tf === Dumping Istio Deployment State for TestClientTracing... 2025-06-24T11:49:55.364777Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8f63931f3-6038-4108-9a15-a190aa1a05d3 2025-06-24T11:49:55.365004Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1135s 2025-06-24T11:49:55.365674Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:55.484382Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8059638cb-dc0f-4053-bdbf-ed73bcffb2e2 2025-06-24T11:49:55.484624Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2331s 2025-06-24T11:49:55.485219Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:55.593653Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8bfdf920e-dabe-4f44-9010-9a555b6abca2 2025-06-24T11:49:55.593826Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3423s 2025-06-24T11:49:55.594436Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestClientTracing (151.00s) FAIL tests/integration/telemetry/tracing/zipkin.TestClientTracing (151.00s) === RUN TestServerTracing/primary-0 2025-06-24T11:49:55.676123Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:50:37.051181Z error port forward failed: lost connection to pod 2025-06-24T11:50:37.513271Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:50:37.557539Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:51:43.562262Z error port forward failed: lost connection to pod 2025-06-24T11:51:44.013486Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:51:44.051878Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T11:52:25.677191Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.00106361s)' === 2025-06-24T11:52:25.677317Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-06-24T11:52:25.677307Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-06-24T11:52:25.883162Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8428962d5-f0ba-440c-93f2-216f54da4408 2025-06-24T11:52:25.883401Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2056s 2025-06-24T11:52:25.884045Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:25.975714Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8eddb8848-8412-4f84-9c97-cf93def069be 2025-06-24T11:52:25.975954Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2982s 2025-06-24T11:52:25.976575Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:26.071013Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8758b089a-0fb6-4a3d-9f73-b38dab89e19d 2025-06-24T11:52:26.071182Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3934s 2025-06-24T11:52:26.071806Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing/primary-0 (150.64s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing/primary-0 (150.64s) === RUN TestServerTracing 2025-06-24T11:49:55.675943Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-06-24T11:52:26.313227Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.637279688s)' === 2025-06-24T11:52:26.313314Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-06-24T11:52:26.313309Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-06-24T11:52:26.416636Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8be1669f8-4a9d-49a3-a648-f9e593e0ea2c 2025-06-24T11:52:26.416944Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1035s 2025-06-24T11:52:26.417480Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:26.501769Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/821bd9aad-9c0f-47ce-9034-947fb021ef5b 2025-06-24T11:52:26.501935Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1884s 2025-06-24T11:52:26.502438Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:26.586102Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8bfeaae26-1b99-4517-8e96-e65d775ca37a 2025-06-24T11:52:26.586269Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2727s 2025-06-24T11:52:26.586734Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing (151.19s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing (151.19s) 2025-06-24T11:52:26.865964Z info tf === FAILED: Test Run: 'telemetry_tracing_zipkin' (exitCode: 1) === 2025-06-24T11:52:26.866030Z info tf === Suite "telemetry_tracing_zipkin" run time: 5m15.246652172s === 2025-06-24T11:52:26.866044Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/trace.yaml 2025-06-24T11:52:26.866995Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:52:26.867000Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:52:27.034797Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/830bb9997-c541-47a7-bde4-f0b90dafbfd9 2025-06-24T11:52:27.035045Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1678s 2025-06-24T11:52:27.035658Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:27.114892Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8974f4949-2b09-4ab1-aa04-4772f827ee8c 2025-06-24T11:52:27.115094Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2478s 2025-06-24T11:52:27.115606Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:27.197081Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8e8931835-292c-4f97-aa9c-eb936f23070b 2025-06-24T11:52:27.197267Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3300s 2025-06-24T11:52:27.197726Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:27.759198Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:52:27.759306Z info tf === SUCCEEDED: Cleanup Istio in 94.627µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin DONE 4 tests, 4 failures in 331.811s exec: [go test -json -test.run=^TestClientTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3470672 2025-06-24T11:52:38.187420Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:52:38.187462Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:52:38.214682Z info tf === Test Framework Settings === 2025-06-24T11:52:38.214799Z info tf TestID: telemetry_tracing_zipkin RunID: 7af99eb8-178d-4cc3-96aa-4878941f68a0 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T11:52:38.214807Z info tf =============================== 2025-06-24T11:52:38.215087Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-7af99e 2025-06-24T11:52:38.215221Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T11:52:38.215237Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T11:52:38.215261Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T11:52:38.215273Z info tf === BEGIN: Building clusters === 2025-06-24T11:52:38.219745Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T11:52:38.219778Z info tf === DONE: Building clusters === 2025-06-24T11:52:38.219798Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-06-24T11:52:38.219832Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:52:38.219837Z info tf === Istio Component Config === 2025-06-24T11:52:38.219885Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T11:52:38.219891Z info tf ================================ 2025-06-24T11:52:38.222334Z info tf skipping deployment as specified in the config 2025-06-24T11:52:38.222348Z info tf === SUCCEEDED: Deploy Istio in 2.513116ms [Suite=telemetry_tracing_zipkin]=== 2025-06-24T11:52:38.222355Z info tf === BEGIN: Create namespace echo === 2025-06-24T11:52:38.234396Z info tf === SUCCEEDED: Create namespace echo in 12.024847ms === 2025-06-24T11:52:38.254247Z info tf === BEGIN: Deploy echo instances === 2025-06-24T11:52:38.260226Z error tf === FAILED: Deploy echo instances === 2025-06-24T11:52:38.260241Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-7af99e/_suite_context/Service.1669999334.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T11:52:38.260272Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:52:38.260328Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:52:38.360429Z error klog an error occurred forwarding 37913 -> 15000: error forwarding port 15000 to pod 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7, uid : port forward into network namespace "/var/run/netns/49b9a89f-13a8-4bd8-8bc5-720cd893fc3e": failed to connect to localhost:15000 inside namespace 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:52:38.362052Z error port forward failed: lost connection to pod 2025-06-24T11:52:38.362056Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:37913/config_dump?include_eds=true": EOF 2025-06-24T11:52:38.362986Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:37913/clusters": dial tcp 127.0.0.1:37913: connect: connection refused 2025-06-24T11:52:38.368075Z error klog an error occurred forwarding 40659 -> 15000: error forwarding port 15000 to pod a9f12a75c7107663cd7f784680804715112515edc4d9cf6a8d0aea16c2947bc5, uid : port forward into network namespace "/var/run/netns/b0e10ffd-e1a2-4059-90ee-a20e291e06f9": failed to connect to localhost:15000 inside namespace a9f12a75c7107663cd7f784680804715112515edc4d9cf6a8d0aea16c2947bc5: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:52:38.370015Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-fdtnp for: Get "http://localhost:40659/config_dump?include_eds=true": EOF 2025-06-24T11:52:38.370047Z error port forward failed: lost connection to pod 2025-06-24T11:52:38.370853Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-fdtnp for: Get "http://localhost:40659/clusters": dial tcp 127.0.0.1:40659: connect: connection refused 2025-06-24T11:52:38.372023Z error klog an error occurred forwarding 35057 -> 15020: error forwarding port 15020 to pod 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7, uid : port forward into network namespace "/var/run/netns/49b9a89f-13a8-4bd8-8bc5-720cd893fc3e": failed to connect to localhost:15020 inside namespace 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:52:38.372981Z error klog an error occurred forwarding 46343 -> 15020: error forwarding port 15020 to pod a9f12a75c7107663cd7f784680804715112515edc4d9cf6a8d0aea16c2947bc5, uid : port forward into network namespace "/var/run/netns/b0e10ffd-e1a2-4059-90ee-a20e291e06f9": failed to connect to localhost:15020 inside namespace a9f12a75c7107663cd7f784680804715112515edc4d9cf6a8d0aea16c2947bc5: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:52:38.373725Z error port forward failed: lost connection to pod 2025-06-24T11:52:38.373793Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:35057/debug/ndsz": EOF 2025-06-24T11:52:38.374577Z error port forward failed: lost connection to pod 2025-06-24T11:52:38.374595Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:35057/stats/prometheus": dial tcp 127.0.0.1:35057: connect: connection refused 2025-06-24T11:52:38.374607Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-fdtnp for: Get "http://localhost:46343/debug/ndsz": EOF 2025-06-24T11:52:38.375272Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-fdtnp for: Get "http://localhost:46343/stats/prometheus": dial tcp 127.0.0.1:46343: connect: connection refused 2025-06-24T11:52:38.392689Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:52:33Z/10e01b7150-f1c1-48b5-ad27-41e0f2f1288a 2025-06-24T11:52:38.392808Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1323s 2025-06-24T11:52:38.393489Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:38.494642Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:52:33Z/100e852b3b-afe4-4424-8532-8f4dc4d20ebd 2025-06-24T11:52:38.494735Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2342s 2025-06-24T11:52:38.495262Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:38.596652Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:52:33Z/1073ef657c-fc31-4342-bcde-7d4c17dd4954 2025-06-24T11:52:38.596748Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3362s 2025-06-24T11:52:38.597223Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:38.840357Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-7af99e/_suite_context/Service.1669999334.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T11:52:38.840403Z info tf === FAILED: Setup: 'telemetry_tracing_zipkin' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-7af99e/_suite_context/Service.1669999334.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T11:52:38.840409Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-7af99e/_suite_context/Service.1669999334.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T11:52:38.840432Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:52:38.840440Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:52:38.906083Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-fdtnp_echo") 2025-06-24T11:52:38.906851Z error klog an error occurred forwarding 35921 -> 15020: error forwarding port 15020 to pod 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7, uid : port forward into network namespace "/var/run/netns/49b9a89f-13a8-4bd8-8bc5-720cd893fc3e": failed to connect to localhost:15020 inside namespace 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:52:38.907309Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-fdtnp_echo") 2025-06-24T11:52:38.908365Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:35921/debug/ndsz": EOF 2025-06-24T11:52:38.908397Z error port forward failed: lost connection to pod 2025-06-24T11:52:38.909024Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:35921/stats/prometheus": dial tcp 127.0.0.1:35921: connect: connection refused 2025-06-24T11:52:38.940477Z error klog an error occurred forwarding 34923 -> 15000: error forwarding port 15000 to pod 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7, uid : port forward into network namespace "/var/run/netns/49b9a89f-13a8-4bd8-8bc5-720cd893fc3e": failed to connect to localhost:15000 inside namespace 8d5c449bd4f801d684bdc685a7fee13a3f0e7cfef74726c7a246f0867b7558f7: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:52:38.942127Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:34923/config_dump?include_eds=true": EOF 2025-06-24T11:52:38.942161Z error port forward failed: lost connection to pod 2025-06-24T11:52:38.942866Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-pvhlk for: Get "http://localhost:34923/clusters": dial tcp 127.0.0.1:34923: connect: connection refused 2025-06-24T11:52:38.973361Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T11:52:38.973362Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T11:52:38.981387Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:52:33Z/103f5a645e-e4dd-4de2-b2d6-30156b797e41 2025-06-24T11:52:38.981477Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1409s 2025-06-24T11:52:38.982045Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:39.081509Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:52:33Z/10ddeab078-bd87-44fe-825f-b53bcfe2995e 2025-06-24T11:52:39.082169Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2415s 2025-06-24T11:52:39.083471Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:39.164918Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:52:33Z/10dce03ff8-0a2a-4525-afab-81c12fcf870d 2025-06-24T11:52:39.165110Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3244s 2025-06-24T11:52:39.165526Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:39.433783Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-pvhlk/app for: pods "server-v1-7d889f5bc4-pvhlk" not found 2025-06-24T11:52:39.453749Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-pvhlk/istio-proxy for: pods "server-v1-7d889f5bc4-pvhlk" not found 2025-06-24T11:52:39.470212Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-pvhlk/istio-validation for: pods "server-v1-7d889f5bc4-pvhlk" not found 2025-06-24T11:52:39.473783Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:52:39.473808Z info tf === SUCCEEDED: Cleanup Istio in 20.485µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin exec: [go test -json -test.run=^TestServerTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3472131 2025-06-24T11:52:51.093543Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:52:51.093601Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:52:51.122757Z info tf === Test Framework Settings === 2025-06-24T11:52:51.122852Z info tf TestID: telemetry_tracing_zipkin RunID: 5d29761f-51b2-4383-90bc-07546062a93c NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T11:52:51.122860Z info tf =============================== 2025-06-24T11:52:51.123023Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-5d2976 2025-06-24T11:52:51.123113Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T11:52:51.123125Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T11:52:51.123143Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T11:52:51.123150Z info tf === BEGIN: Building clusters === 2025-06-24T11:52:51.127352Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T11:52:51.127399Z info tf === DONE: Building clusters === 2025-06-24T11:52:51.127420Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-06-24T11:52:51.127449Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:52:51.127456Z info tf === Istio Component Config === 2025-06-24T11:52:51.127495Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T11:52:51.127501Z info tf ================================ 2025-06-24T11:52:51.130053Z info tf skipping deployment as specified in the config 2025-06-24T11:52:51.130066Z info tf === SUCCEEDED: Deploy Istio in 2.613882ms [Suite=telemetry_tracing_zipkin]=== 2025-06-24T11:52:51.130073Z info tf === BEGIN: Create namespace echo === 2025-06-24T11:52:51.162455Z info tf === SUCCEEDED: Create namespace echo in 32.357549ms === 2025-06-24T11:52:51.213047Z info tf === BEGIN: Deploy echo instances === 2025-06-24T11:52:51.388049Z info klog Waiting for caches to sync for echo 2025-06-24T11:52:51.390348Z info klog Waiting for caches to sync for echo 2025-06-24T11:52:51.489006Z info klog Caches are synced for echo 2025-06-24T11:52:51.490462Z info klog Caches are synced for echo 2025-06-24T11:52:57.793432Z info tf === SUCCEEDED: Deploy echo instances in 6.580373179s === 2025-06-24T11:52:57.838764Z info tf Checking pods ready... 2025-06-24T11:52:57.838806Z info tf Checking pods ready... 2025-06-24T11:52:57.844708Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Pending (Pending) 2025-06-24T11:52:58.045791Z info tf Checking pods ready... 2025-06-24T11:52:58.045820Z info tf Checking pods ready... 2025-06-24T11:52:58.050181Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Pending (Pending) 2025-06-24T11:52:58.450930Z info tf Checking pods ready... 2025-06-24T11:52:58.451003Z info tf Checking pods ready... 2025-06-24T11:52:58.456045Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Pending (Pending) 2025-06-24T11:52:59.256230Z info tf Checking pods ready... 2025-06-24T11:52:59.256302Z info tf Checking pods ready... 2025-06-24T11:52:59.266504Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Pending (Pending) 2025-06-24T11:53:00.867420Z info tf Checking pods ready... 2025-06-24T11:53:00.867464Z info tf Checking pods ready... 2025-06-24T11:53:00.879388Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Running (container not ready: 'zipkin') 2025-06-24T11:53:04.079794Z info tf Checking pods ready... 2025-06-24T11:53:04.079855Z info tf Checking pods ready... 2025-06-24T11:53:04.084191Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Running (container not ready: 'zipkin') 2025-06-24T11:53:07.284300Z info tf Checking pods ready... 2025-06-24T11:53:07.284344Z info tf Checking pods ready... 2025-06-24T11:53:07.288408Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Running (container not ready: 'zipkin') 2025-06-24T11:53:10.489283Z info tf Checking pods ready... 2025-06-24T11:53:10.489321Z info tf Checking pods ready... 2025-06-24T11:53:10.493047Z info tf [ 0] zipkin-77cbd75c85-g5gc5 Running (Ready) 2025-06-24T11:53:10.580549Z info tf === DONE: Setup: 'telemetry_tracing_zipkin' (19.453112427s) === 2025-06-24T11:53:10.580598Z info tf === BEGIN: Test Run: 'telemetry_tracing_zipkin' === === RUN TestServerTracing/primary-0 2025-06-24T11:53:10.581275Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:54:03.698277Z error port forward failed: lost connection to pod 2025-06-24T11:54:03.978115Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward terminated 2025-06-24T11:54:04.017201Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:54:10.519546Z error port forward failed: lost connection to pod server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:55:10.026274Z error port forward failed: lost connection to pod 2025-06-24T11:55:10.478345Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward terminated 2025-06-24T11:55:10.514900Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T11:55:40.582273Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.000996258s)' === 2025-06-24T11:55:40.582351Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-06-24T11:55:40.582370Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-06-24T11:55:40.820718Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/156564525a-5162-47ca-ab14-c10a9450e161 2025-06-24T11:55:40.820940Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2383s 2025-06-24T11:55:40.821471Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:40.926284Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/159ab936bd-67c6-4a4a-ba7b-3a084fb39110 2025-06-24T11:55:40.926479Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3438s 2025-06-24T11:55:40.927056Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:41.023748Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15cf91eb5d-b8d7-4f47-baae-5303032afd8d 2025-06-24T11:55:41.023903Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4413s 2025-06-24T11:55:41.024472Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing/primary-0 (150.72s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing/primary-0 (re-run 1) (150.72s) === RUN TestServerTracing 2025-06-24T11:53:10.581044Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-06-24T11:55:41.300678Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.71962748s)' === 2025-06-24T11:55:41.300781Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-06-24T11:55:41.300782Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-06-24T11:55:41.408500Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15bcb1b041-95f7-4903-8e56-6171d5f4eb31 2025-06-24T11:55:41.408787Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1078s 2025-06-24T11:55:41.409279Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:41.491635Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15bd48398b-23d8-4dfa-bac6-bb06ce4bb0f1 2025-06-24T11:55:41.491776Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1908s 2025-06-24T11:55:41.492213Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:41.612937Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15fab4e428-94c5-498e-8d51-acfe47807216 2025-06-24T11:55:41.613121Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3121s 2025-06-24T11:55:41.613895Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestServerTracing (151.26s) FAIL tests/integration/telemetry/tracing/zipkin.TestServerTracing (re-run 1) (151.26s) 2025-06-24T11:55:41.841318Z info tf === FAILED: Test Run: 'telemetry_tracing_zipkin' (exitCode: 1) === 2025-06-24T11:55:41.842340Z info tf === Suite "telemetry_tracing_zipkin" run time: 2m50.714917754s === 2025-06-24T11:55:41.842380Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/trace.yaml 2025-06-24T11:55:41.842643Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:55:41.842674Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:55:41.982538Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/1510f14ee0-5ecd-4e88-be23-53e9a07ef60c 2025-06-24T11:55:41.982740Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1399s 2025-06-24T11:55:41.983380Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:42.100811Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/151c587a1a-817e-4ec7-9b75-c471c6cdb2f6 2025-06-24T11:55:42.100957Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2581s 2025-06-24T11:55:42.101381Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:42.198094Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15e2a154ca-4009-4a50-9544-f0dd241eb9fe 2025-06-24T11:55:42.198242Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3553s 2025-06-24T11:55:42.198777Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:42.508238Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:55:42.508350Z info tf === SUCCEEDED: Cleanup Istio in 99.998µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin DONE 2 runs, 6 tests, 6 failures in 526.571s exec: [go test -json -test.run=^TestServerTracing$/^primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/zipkin -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts] go test pid: 3473523 2025-06-24T11:55:53.293569Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:55:53.293622Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T11:55:53.324071Z info tf === Test Framework Settings === 2025-06-24T11:55:53.324159Z info tf TestID: telemetry_tracing_zipkin RunID: 6f2b126c-b755-4733-be87-6651a18125e5 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T11:55:53.324165Z info tf =============================== 2025-06-24T11:55:53.324346Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-6f2b12 2025-06-24T11:55:53.324429Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T11:55:53.324442Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T11:55:53.324461Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T11:55:53.324470Z info tf === BEGIN: Building clusters === 2025-06-24T11:55:53.326896Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T11:55:53.326903Z info tf === DONE: Building clusters === 2025-06-24T11:55:53.326922Z info tf === BEGIN: Setup: 'telemetry_tracing_zipkin' === 2025-06-24T11:55:53.326948Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:55:53.326955Z info tf === Istio Component Config === 2025-06-24T11:55:53.326994Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.proxy.tracer:zipkin global.tag:ibm-p global.variant: meshConfig.enableTracing:true pilot.traceSampling:100.0] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T11:55:53.327000Z info tf ================================ 2025-06-24T11:55:53.329336Z info tf skipping deployment as specified in the config 2025-06-24T11:55:53.329348Z info tf === SUCCEEDED: Deploy Istio in 2.39671ms [Suite=telemetry_tracing_zipkin]=== 2025-06-24T11:55:53.329354Z info tf === BEGIN: Create namespace echo === 2025-06-24T11:55:53.347416Z info tf === SUCCEEDED: Create namespace echo in 18.044485ms === 2025-06-24T11:55:53.374946Z info tf === BEGIN: Deploy echo instances === 2025-06-24T11:55:53.380508Z error tf === FAILED: Deploy echo instances === 2025-06-24T11:55:53.380549Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-6f2b12/_suite_context/Service.2922238603.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T11:55:53.380581Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:55:53.380593Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:55:53.450855Z error klog an error occurred forwarding 34549 -> 15020: error forwarding port 15020 to pod 32b469ec73f26e3a4665bd007a6d76ddd60c2610d8d12f39343a4419f39ca2bc, uid : port forward into network namespace "/var/run/netns/e49f6f18-9cc6-4f3f-a8cc-a76efd39de09": failed to connect to localhost:15020 inside namespace 32b469ec73f26e3a4665bd007a6d76ddd60c2610d8d12f39343a4419f39ca2bc: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:55:53.452766Z error port forward failed: lost connection to pod 2025-06-24T11:55:53.452811Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-4k49l for: Get "http://localhost:34549/debug/ndsz": EOF 2025-06-24T11:55:53.453638Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-4k49l for: Get "http://localhost:34549/stats/prometheus": dial tcp 127.0.0.1:34549: connect: connection refused 2025-06-24T11:55:53.461980Z error klog an error occurred forwarding 45201 -> 15000: error forwarding port 15000 to pod 32b469ec73f26e3a4665bd007a6d76ddd60c2610d8d12f39343a4419f39ca2bc, uid : port forward into network namespace "/var/run/netns/e49f6f18-9cc6-4f3f-a8cc-a76efd39de09": failed to connect to localhost:15000 inside namespace 32b469ec73f26e3a4665bd007a6d76ddd60c2610d8d12f39343a4419f39ca2bc: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:55:53.462110Z error klog an error occurred forwarding 33051 -> 15020: error forwarding port 15020 to pod d82c3efb510069f7c51ef12aad27f4bb5480f5c3d06b9f217e5dbb4e21c28f94, uid : port forward into network namespace "/var/run/netns/3b976b3e-1407-4de9-b57c-008c5e9b5100": failed to connect to localhost:15020 inside namespace d82c3efb510069f7c51ef12aad27f4bb5480f5c3d06b9f217e5dbb4e21c28f94: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:55:53.463507Z error port forward failed: lost connection to pod 2025-06-24T11:55:53.463545Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-4k49l for: Get "http://localhost:45201/config_dump?include_eds=true": EOF 2025-06-24T11:55:53.463763Z error port forward failed: lost connection to pod 2025-06-24T11:55:53.463763Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-xj7bw for: Get "http://localhost:33051/debug/ndsz": EOF 2025-06-24T11:55:53.467145Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-xj7bw for: Get "http://localhost:33051/stats/prometheus": dial tcp 127.0.0.1:33051: connect: connection refused 2025-06-24T11:55:53.467186Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-4k49l for: Get "http://localhost:45201/clusters": dial tcp 127.0.0.1:45201: connect: connection refused 2025-06-24T11:55:53.467467Z error klog an error occurred forwarding 46535 -> 15000: error forwarding port 15000 to pod d82c3efb510069f7c51ef12aad27f4bb5480f5c3d06b9f217e5dbb4e21c28f94, uid : port forward into network namespace "/var/run/netns/3b976b3e-1407-4de9-b57c-008c5e9b5100": failed to connect to localhost:15000 inside namespace d82c3efb510069f7c51ef12aad27f4bb5480f5c3d06b9f217e5dbb4e21c28f94: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T11:55:53.468949Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-xj7bw for: Get "http://localhost:46535/config_dump?include_eds=true": EOF 2025-06-24T11:55:53.468995Z error port forward failed: lost connection to pod 2025-06-24T11:55:53.469660Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-xj7bw for: Get "http://localhost:46535/clusters": dial tcp 127.0.0.1:46535: connect: connection refused 2025-06-24T11:55:53.505012Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:55:48Z/17a1a9c7a3-47db-4f35-92c0-3d6253c5cc61 2025-06-24T11:55:53.505266Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1244s 2025-06-24T11:55:53.505807Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:53.607939Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:55:48Z/179ab0a501-1160-48d7-8ee9-4d9bfa6f8e52 2025-06-24T11:55:53.608093Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2272s 2025-06-24T11:55:53.608653Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:53.716485Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:55:48Z/17cac98168-a82c-471b-a7e6-1034431032e9 2025-06-24T11:55:53.716715Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3358s 2025-06-24T11:55:53.717381Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:53.823838Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-6f2b12/_suite_context/Service.2922238603.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T11:55:53.823913Z info tf === FAILED: Setup: 'telemetry_tracing_zipkin' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-6f2b12/_suite_context/Service.2922238603.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T11:55:53.823920Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-zipkin/artifacts/telemetry-tracing-zipkin-6f2b12/_suite_context/Service.2922238603.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T11:55:53.823950Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:55:53.823957Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_zipkin)]... 2025-06-24T11:55:53.885825Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-4k49l_echo") 2025-06-24T11:55:53.889404Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-4k49l_echo") 2025-06-24T11:55:53.943789Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:55:48Z/171075db51-2485-434c-bc7b-872b5fff22d2 2025-06-24T11:55:53.943988Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1198s 2025-06-24T11:55:53.944578Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:53.949975Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T11:55:53.952353Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T11:55:54.056014Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:55:48Z/17a9f8b8f5-2dcf-4c18-8bdd-73ccb681bc9e 2025-06-24T11:55:54.056150Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2319s 2025-06-24T11:55:54.056622Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:54.152628Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:55:48Z/17f4c83719-b4b7-444d-89c6-6f5c4c79b450 2025-06-24T11:55:54.152807Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3286s 2025-06-24T11:55:54.153386Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:54.232455Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_zipkin] === 2025-06-24T11:55:54.232511Z info tf === SUCCEEDED: Cleanup Istio in 47.923µs [Suite=telemetry_tracing_zipkin] === FAIL tests/integration/telemetry/tracing/zipkin === Failed === FAIL: tests/integration/telemetry/tracing/zipkin TestClientTracing/primary-0 (150.58s) 2025-06-24T11:47:24.675741Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0]' === client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... 2025-06-24T11:48:24.245020Z error port forward failed: lost connection to pod 2025-06-24T11:48:24.512970Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:48:24.549163Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success 2025-06-24T11:48:24.637345Z error port forward failed: lost connection to pod client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... 2025-06-24T11:49:30.552698Z error port forward failed: lost connection to pod 2025-06-24T11:49:31.013657Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:49:31.046295Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:50: Sending from cluster-0... client_tracing_test.go:44: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T11:49:54.677083Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing/primary-0] (2m30.001336075s)' === 2025-06-24T11:49:54.677222Z error tf === Dumping Istio Deployment State for TestClientTracing/primary-0... 2025-06-24T11:49:54.677241Z error tf === Dumping Namespace echo State for TestClientTracing/primary-0... 2025-06-24T11:49:54.906055Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8b41ce590-d236-463a-bbf0-9ee8ea7235a3 2025-06-24T11:49:54.906321Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2288s 2025-06-24T11:49:54.906808Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:54.988377Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8893696be-8faa-4c67-bf5f-622a94e77b61 2025-06-24T11:49:54.988482Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3109s 2025-06-24T11:49:54.989120Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:55.081790Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/88a60e234-41a2-4ee2-b1ca-61343613d67a 2025-06-24T11:49:55.081964Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4043s 2025-06-24T11:49:55.082555Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestClientTracing (151.00s) 2025-06-24T11:47:24.675555Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestClientTracing]' === 2025-06-24T11:49:55.251281Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestClientTracing] (2m30.575722903s)' === 2025-06-24T11:49:55.251363Z error tf === Dumping Namespace echo State for TestClientTracing... 2025-06-24T11:49:55.251367Z error tf === Dumping Istio Deployment State for TestClientTracing... 2025-06-24T11:49:55.364777Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8f63931f3-6038-4108-9a15-a190aa1a05d3 2025-06-24T11:49:55.365004Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1135s 2025-06-24T11:49:55.365674Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:55.484382Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8059638cb-dc0f-4053-bdbf-ed73bcffb2e2 2025-06-24T11:49:55.484624Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2331s 2025-06-24T11:49:55.485219Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:49:55.593653Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8bfdf920e-dabe-4f44-9010-9a555b6abca2 2025-06-24T11:49:55.593826Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3423s 2025-06-24T11:49:55.594436Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing/primary-0 (150.64s) 2025-06-24T11:49:55.676123Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:50:37.051181Z error port forward failed: lost connection to pod 2025-06-24T11:50:37.513271Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:50:37.557539Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:51:43.562262Z error port forward failed: lost connection to pod 2025-06-24T11:51:44.013486Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward terminated 2025-06-24T11:51:44.051878Z warn pod: echo/server-v1-7d889f5bc4-pvhlk port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T11:52:25.677191Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.00106361s)' === 2025-06-24T11:52:25.677317Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-06-24T11:52:25.677307Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-06-24T11:52:25.883162Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8428962d5-f0ba-440c-93f2-216f54da4408 2025-06-24T11:52:25.883401Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2056s 2025-06-24T11:52:25.884045Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:25.975714Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8eddb8848-8412-4f84-9c97-cf93def069be 2025-06-24T11:52:25.975954Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2982s 2025-06-24T11:52:25.976575Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:26.071013Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8758b089a-0fb6-4a3d-9f73-b38dab89e19d 2025-06-24T11:52:26.071182Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3934s 2025-06-24T11:52:26.071806Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing (151.19s) 2025-06-24T11:49:55.675943Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-06-24T11:52:26.313227Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.637279688s)' === 2025-06-24T11:52:26.313314Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-06-24T11:52:26.313309Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-06-24T11:52:26.416636Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8be1669f8-4a9d-49a3-a648-f9e593e0ea2c 2025-06-24T11:52:26.416944Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1035s 2025-06-24T11:52:26.417480Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:26.501769Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/821bd9aad-9c0f-47ce-9034-947fb021ef5b 2025-06-24T11:52:26.501935Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1884s 2025-06-24T11:52:26.502438Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:52:26.586102Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:47:25Z/8bfeaae26-1b99-4517-8e96-e65d775ca37a 2025-06-24T11:52:26.586269Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2727s 2025-06-24T11:52:26.586734Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing/primary-0 (re-run 1) (150.72s) 2025-06-24T11:53:10.581275Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0]' === server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:54:03.698277Z error port forward failed: lost connection to pod 2025-06-24T11:54:03.978115Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward terminated 2025-06-24T11:54:04.017201Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:54:10.519546Z error port forward failed: lost connection to pod server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... 2025-06-24T11:55:10.026274Z error port forward failed: lost connection to pod 2025-06-24T11:55:10.478345Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward terminated 2025-06-24T11:55:10.514900Z warn pod: echo/server-v1-7d889f5bc4-xj7bw port forward reconnect success server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:43: Sending from cluster-0... server_tracing_test.go:42: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T11:55:40.582273Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing/primary-0] (2m30.000996258s)' === 2025-06-24T11:55:40.582351Z error tf === Dumping Namespace echo State for TestServerTracing/primary-0... 2025-06-24T11:55:40.582370Z error tf === Dumping Istio Deployment State for TestServerTracing/primary-0... 2025-06-24T11:55:40.820718Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/156564525a-5162-47ca-ab14-c10a9450e161 2025-06-24T11:55:40.820940Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2383s 2025-06-24T11:55:40.821471Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:40.926284Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/159ab936bd-67c6-4a4a-ba7b-3a084fb39110 2025-06-24T11:55:40.926479Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3438s 2025-06-24T11:55:40.927056Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:41.023748Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15cf91eb5d-b8d7-4f47-baae-5303032afd8d 2025-06-24T11:55:41.023903Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4413s 2025-06-24T11:55:41.024472Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/zipkin TestServerTracing (re-run 1) (151.26s) 2025-06-24T11:53:10.581044Z info tf === BEGIN: Test: 'telemetry_tracing_zipkin[TestServerTracing]' === 2025-06-24T11:55:41.300678Z info tf === DONE (failed): Test: 'telemetry_tracing_zipkin[TestServerTracing] (2m30.71962748s)' === 2025-06-24T11:55:41.300781Z error tf === Dumping Namespace echo State for TestServerTracing... 2025-06-24T11:55:41.300782Z error tf === Dumping Istio Deployment State for TestServerTracing... 2025-06-24T11:55:41.408500Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15bcb1b041-95f7-4903-8e56-6171d5f4eb31 2025-06-24T11:55:41.408787Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1078s 2025-06-24T11:55:41.409279Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:41.491635Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15bd48398b-23d8-4dfa-bac6-bb06ce4bb0f1 2025-06-24T11:55:41.491776Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1908s 2025-06-24T11:55:41.492213Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T11:55:41.612937Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T11:53:11Z/15fab4e428-94c5-498e-8d51-acfe47807216 2025-06-24T11:55:41.613121Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3121s 2025-06-24T11:55:41.613895Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 3 runs, 6 tests, 6 failures in 538.256s exec: go version [root@cpt-ppc-059 istio]#