[root@cpt-ppc-059 istio]# clear [root@cpt-ppc-059 istio]# gotestsum -f testname --junitfile-project-name istio --junitfile /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/junit-telemetry-tracing-otelcollector.xml --rerun-fails --packages=./tests/integration/telemetry/tracing/otelcollector --rerun-fails-max-failures=30 --debug -- -tags=integ -timeout 180m -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts exec: [go test -json -tags=integ -timeout 180m ./tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3437079 2025-06-24T10:23:15.151027Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:23:15.151071Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:23:15.184042Z info tf === Test Framework Settings === 2025-06-24T10:23:15.184155Z info tf TestID: telemetry_tracing_otelcollector RunID: f77af289-4dac-4bea-b593-6fd2ad9b4631 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:23:15.184163Z info tf =============================== 2025-06-24T10:23:15.185309Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:23:15.185701Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:23:15.185716Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:23:15.185737Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:23:15.185747Z info tf === BEGIN: Building clusters === 2025-06-24T10:23:15.188843Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:23:15.188854Z info tf === DONE: Building clusters === 2025-06-24T10:23:15.188877Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:23:15.188904Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:23:15.188912Z info tf === Istio Component Config === 2025-06-24T10:23:15.188962Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:23:15.188970Z info tf ================================ 2025-06-24T10:23:15.196763Z info tf skipping deployment as specified in the config 2025-06-24T10:23:15.196833Z info tf === SUCCEEDED: Deploy Istio in 7.921909ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:23:15.196843Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:23:15.225901Z info tf === SUCCEEDED: Create namespace echo in 29.054974ms === 2025-06-24T10:23:15.282850Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:23:15.471928Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:15.517490Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:15.572634Z info klog Caches are synced for echo 2025-06-24T10:23:15.618480Z info klog Caches are synced for echo 2025-06-24T10:23:29.927063Z info tf === SUCCEEDED: Deploy echo instances in 14.644190892s === 2025-06-24T10:23:29.985046Z info tf Checking pods ready... 2025-06-24T10:23:29.985080Z info tf Checking pods ready... 2025-06-24T10:23:29.990767Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:30.191262Z info tf Checking pods ready... 2025-06-24T10:23:30.191328Z info tf Checking pods ready... 2025-06-24T10:23:30.196644Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:30.597584Z info tf Checking pods ready... 2025-06-24T10:23:30.597611Z info tf Checking pods ready... 2025-06-24T10:23:30.603433Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:31.404079Z info tf Checking pods ready... 2025-06-24T10:23:31.404125Z info tf Checking pods ready... 2025-06-24T10:23:31.410472Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:33.011215Z info tf Checking pods ready... 2025-06-24T10:23:33.011256Z info tf Checking pods ready... 2025-06-24T10:23:33.017352Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:36.217947Z info tf Checking pods ready... 2025-06-24T10:23:36.218040Z info tf Checking pods ready... 2025-06-24T10:23:36.222765Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:39.423018Z info tf Checking pods ready... 2025-06-24T10:23:39.423064Z info tf Checking pods ready... 2025-06-24T10:23:39.427982Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:42.628999Z info tf Checking pods ready... 2025-06-24T10:23:42.629033Z info tf Checking pods ready... 2025-06-24T10:23:42.634322Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:45.835139Z info tf Checking pods ready... 2025-06-24T10:23:45.835175Z info tf Checking pods ready... 2025-06-24T10:23:45.839309Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:49.039644Z info tf Checking pods ready... 2025-06-24T10:23:49.039678Z info tf Checking pods ready... 2025-06-24T10:23:49.044669Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:52.244807Z info tf Checking pods ready... 2025-06-24T10:23:52.244852Z info tf Checking pods ready... 2025-06-24T10:23:52.249452Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:55.450352Z info tf Checking pods ready... 2025-06-24T10:23:55.450403Z info tf Checking pods ready... 2025-06-24T10:23:55.454834Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:23:58.655842Z info tf Checking pods ready... 2025-06-24T10:23:58.655876Z info tf Checking pods ready... 2025-06-24T10:23:58.660273Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:24:01.860967Z info tf Checking pods ready... 2025-06-24T10:24:01.861009Z info tf Checking pods ready... 2025-06-24T10:24:01.865404Z info tf [ 0] zipkin-77cbd75c85-hjlpb Pending (Pending) 2025-06-24T10:24:05.066438Z info tf Checking pods ready... 2025-06-24T10:24:05.066469Z info tf Checking pods ready... 2025-06-24T10:24:05.072335Z info tf [ 0] zipkin-77cbd75c85-hjlpb Running (container not ready: 'zipkin') 2025-06-24T10:24:08.272945Z info tf Checking pods ready... 2025-06-24T10:24:08.273003Z info tf Checking pods ready... 2025-06-24T10:24:08.278101Z info tf [ 0] zipkin-77cbd75c85-hjlpb Running (container not ready: 'zipkin') 2025-06-24T10:24:11.479152Z info tf Checking pods ready... 2025-06-24T10:24:11.479204Z info tf Checking pods ready... 2025-06-24T10:24:11.483934Z info tf [ 0] zipkin-77cbd75c85-hjlpb Running (Ready) 2025-06-24T10:24:11.585810Z info tf Checking pods ready... 2025-06-24T10:24:11.585845Z info tf Checking pods ready... 2025-06-24T10:24:11.589311Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T10:24:11.789959Z info tf Checking pods ready... 2025-06-24T10:24:11.790000Z info tf Checking pods ready... 2025-06-24T10:24:11.798594Z info tf [ 0] opentelemetry-collector-854b84c844-hhwcr Pending (Pending) 2025-06-24T10:24:12.199618Z info tf Checking pods ready... 2025-06-24T10:24:12.199735Z info tf Checking pods ready... 2025-06-24T10:24:12.206122Z info tf [ 0] opentelemetry-collector-854b84c844-hhwcr Pending (Pending) 2025-06-24T10:24:13.006732Z info tf Checking pods ready... 2025-06-24T10:24:13.006781Z info tf Checking pods ready... 2025-06-24T10:24:13.011462Z info tf [ 0] opentelemetry-collector-854b84c844-hhwcr Pending (Pending) 2025-06-24T10:24:14.611994Z info tf Checking pods ready... 2025-06-24T10:24:14.612026Z info tf Checking pods ready... 2025-06-24T10:24:14.615006Z info tf [ 0] opentelemetry-collector-854b84c844-hhwcr Pending (Pending) 2025-06-24T10:24:17.815813Z info tf Checking pods ready... 2025-06-24T10:24:17.815847Z info tf Checking pods ready... 2025-06-24T10:24:17.819594Z info tf [ 0] opentelemetry-collector-854b84c844-hhwcr Pending (Pending) 2025-06-24T10:24:21.020352Z info tf Checking pods ready... 2025-06-24T10:24:21.020403Z info tf Checking pods ready... 2025-06-24T10:24:21.025611Z info tf [ 0] opentelemetry-collector-854b84c844-hhwcr Pending (Pending) 2025-06-24T10:24:24.225974Z info tf Checking pods ready... 2025-06-24T10:24:24.226009Z info tf Checking pods ready... 2025-06-24T10:24:24.230441Z info tf [ 0] opentelemetry-collector-854b84c844-hhwcr Running (Ready) 2025-06-24T10:24:24.268674Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (1m9.079784879s) === 2025-06-24T10:24:24.268700Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 2025-06-24T10:24:24.281983Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:26:54.283120Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.001130896s)' === 2025-06-24T10:26:54.283187Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T10:26:54.283194Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T10:26:54.398185Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/9465ab84d-21c1-4381-967d-ba7581229b1a 2025-06-24T10:26:54.398629Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1149s 2025-06-24T10:26:54.398907Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:54.483642Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/91ba2bb76-12c7-46ca-941e-88076948453f 2025-06-24T10:26:54.483756Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2000s 2025-06-24T10:26:54.484311Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:54.572648Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/92a8a4891-f463-44d5-b4a9-6ee6e9dc1570 2025-06-24T10:26:54.572805Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2890s 2025-06-24T10:26:54.573285Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (150.62s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (150.62s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter 2025-06-24T10:24:24.268997Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T10:26:54.898091Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m30.629090776s)' === 2025-06-24T10:26:54.898164Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T10:26:54.898167Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T10:26:55.003003Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/9e4e438b6-96dc-43bd-9530-f416468b5286 2025-06-24T10:26:55.003330Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1050s 2025-06-24T10:26:55.003824Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:55.100949Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/93ac9ea10-b906-49de-9fd2-bfabfb62ee70 2025-06-24T10:26:55.101108Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2027s 2025-06-24T10:26:55.101557Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:55.187909Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/91367b36a-05bc-4838-8e5f-c0b0a6da7f07 2025-06-24T10:26:55.188175Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2898s 2025-06-24T10:26:55.188766Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter (151.11s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter (151.11s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 2025-06-24T10:26:55.396624Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:29:25.396941Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000314196s)' === 2025-06-24T10:29:25.396985Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:29:25.397182Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:29:25.620678Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10b0231a58-bdd0-4f8a-a2fb-0af6042a4f9f 2025-06-24T10:29:25.621064Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2231s 2025-06-24T10:29:25.621516Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:25.706044Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10041866f4-3a00-42c8-ab36-e38b23a8e176 2025-06-24T10:29:25.706187Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3082s 2025-06-24T10:29:25.706856Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:25.791038Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/1088901462-5d2f-4384-a2a4-ca839b46ab1a 2025-06-24T10:29:25.791219Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3933s 2025-06-24T10:29:25.791777Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (150.60s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (150.60s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter 2025-06-24T10:26:55.380398Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T10:29:25.999641Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.619239947s)' === 2025-06-24T10:29:25.999741Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:29:25.999750Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:29:26.228611Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10d5c5ecf4-b095-444e-8ac7-c372733689e5 2025-06-24T10:29:26.228932Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2290s 2025-06-24T10:29:26.229527Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:26.321146Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10cbe047d8-c021-40a9-ac90-6e68caf748a5 2025-06-24T10:29:26.321435Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3215s 2025-06-24T10:29:26.321878Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:26.404807Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10d9fb423a-aea5-4760-bb1d-5a87be777708 2025-06-24T10:29:26.404967Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4050s 2025-06-24T10:29:26.405479Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter (151.68s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter (151.68s) === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 2025-06-24T10:29:27.077662Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:31:57.078651Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.000984655s)' === 2025-06-24T10:31:57.078795Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:31:57.078807Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:31:57.247984Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11033878dd-bb37-4d18-8109-7418280443c1 2025-06-24T10:31:57.248278Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1657s 2025-06-24T10:31:57.249007Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:57.536615Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/111b5777a1-9f22-45ba-a717-93dfbc5b4e79 2025-06-24T10:31:57.536758Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4541s 2025-06-24T10:31:57.537531Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:57.648310Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11fb606a1c-ed1b-4841-9b44-344fcbad8c15 2025-06-24T10:31:57.648456Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5658s 2025-06-24T10:31:57.649135Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (151.29s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (151.29s) === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors 2025-06-24T10:29:27.065836Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T10:31:58.364632Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m31.298795927s)' === 2025-06-24T10:31:58.364672Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:31:58.364678Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:31:58.486489Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11974b34de-640c-461b-aae0-a94bf7aa98ca 2025-06-24T10:31:58.486828Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1211s 2025-06-24T10:31:58.487362Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:58.579387Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11cd2aac99-8dbd-4764-ba69-6fa2546a0d37 2025-06-24T10:31:58.579547Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2138s 2025-06-24T10:31:58.580147Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:58.677749Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/111c026da9-cd8a-4882-9a42-14d4c6659ac2 2025-06-24T10:31:58.677969Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3121s 2025-06-24T10:31:58.678569Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors (152.45s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors (152.45s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 2025-06-24T10:31:59.532805Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:34:29.533272Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000465483s)' === 2025-06-24T10:34:29.533352Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:34:29.533345Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:34:29.813266Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/127be79e69-453c-4852-bf46-10964075c52f 2025-06-24T10:34:29.813688Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2795s 2025-06-24T10:34:29.814176Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:29.913795Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/121f58ef3e-771c-4cbe-ac10-d0d52afb66b4 2025-06-24T10:34:29.914008Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3799s 2025-06-24T10:34:29.914636Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:30.031278Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/126917b8ad-1e62-4bad-96a2-cdaa5ac8c8ca 2025-06-24T10:34:30.031475Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4973s 2025-06-24T10:34:30.032082Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (151.31s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (151.31s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata 2025-06-24T10:31:59.519056Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T10:34:30.846714Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m31.327651868s)' === 2025-06-24T10:34:30.846866Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:34:30.846863Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:34:30.972360Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/12e3d0fd95-20db-4511-8384-45a15c18c5cb 2025-06-24T10:34:30.972565Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1254s 2025-06-24T10:34:30.973315Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:31.074305Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/12e0ef809f-cd41-42d8-939a-33a7ad1ef1e8 2025-06-24T10:34:31.074421Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2273s 2025-06-24T10:34:31.074934Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:31.171566Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/12de6cc5d9-21cc-4993-8979-cdc54c243162 2025-06-24T10:34:31.172019Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3248s 2025-06-24T10:34:31.172323Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (152.77s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (152.77s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T10:24:24.268913Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:24:35.351298Z error port forward failed: lost connection to pod 2025-06-24T10:24:35.588028Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:24:35.620946Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:25:11.512353Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:25:41.625366Z error port forward failed: lost connection to pod 2025-06-24T10:25:42.088199Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:25:42.155637Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:26:48.160566Z error port forward failed: lost connection to pod 2025-06-24T10:26:48.587320Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:26:48.685981Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:27:54.690429Z error port forward failed: lost connection to pod 2025-06-24T10:27:55.088305Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:27:55.125242Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:29:01.130102Z error port forward failed: lost connection to pod 2025-06-24T10:29:01.587651Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:29:01.625602Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:30:07.634263Z error port forward failed: lost connection to pod 2025-06-24T10:30:08.088253Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:30:08.134770Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:31:14.138454Z error port forward failed: lost connection to pod 2025-06-24T10:31:14.588175Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:31:14.620522Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:32:20.625308Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:34:32.293470Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (10m8.024554033s)' === 2025-06-24T10:34:32.293548Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:34:32.293546Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:34:32.422810Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/13c405b575-90e8-4d82-bd1c-35727d90b98a 2025-06-24T10:34:32.423211Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1288s 2025-06-24T10:34:32.423662Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:32.525598Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/136f39b33d-ea4f-4c43-aa3f-6fff094214ca 2025-06-24T10:34:32.525774Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2313s 2025-06-24T10:34:32.526508Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:32.625256Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/13d0793073-c611-4f1a-ae25-f459f036057e 2025-06-24T10:34:32.625439Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3309s 2025-06-24T10:34:32.625965Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (609.58s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (609.58s) 2025-06-24T10:34:33.849195Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T10:34:33.849265Z info tf === Suite "telemetry_tracing_otelcollector" run time: 11m18.660395746s === 2025-06-24T10:34:33.849281Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T10:34:33.852110Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:34:33.852114Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:34:33.978591Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/13dbd86617-8230-45be-bf48-bbdc2c7390d5 2025-06-24T10:34:33.978916Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1259s 2025-06-24T10:34:33.979358Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:34.069691Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/1336592e2a-44e3-4fe3-98cd-59e0f75de065 2025-06-24T10:34:34.069896Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2168s 2025-06-24T10:34:34.070473Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:34.160527Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/13f4e5eecb-34e3-490d-875f-e86507aef25d 2025-06-24T10:34:34.160779Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3077s 2025-06-24T10:34:34.161561Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:35.482152Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:34:35.482279Z info tf === SUCCEEDED: Cleanup Istio in 112.566µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector DONE 9 tests, 9 failures in 695.335s exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3444065 2025-06-24T10:34:49.702677Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:34:49.702724Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:34:49.735579Z info tf === Test Framework Settings === 2025-06-24T10:34:49.735648Z info tf TestID: telemetry_tracing_otelcollector RunID: f7386ba8-27b8-4217-b4cf-58a58977815a NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:34:49.735653Z info tf =============================== 2025-06-24T10:34:49.735678Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:34:49.735713Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:34:49.735722Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:34:49.735739Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:34:49.735745Z info tf === BEGIN: Building clusters === 2025-06-24T10:34:49.738162Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:34:49.738171Z info tf === DONE: Building clusters === 2025-06-24T10:34:49.738189Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:34:49.738692Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:34:49.738708Z info tf === Istio Component Config === 2025-06-24T10:34:49.738800Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:34:49.738815Z info tf ================================ 2025-06-24T10:34:49.742308Z info tf skipping deployment as specified in the config 2025-06-24T10:34:49.742321Z info tf === SUCCEEDED: Deploy Istio in 3.627377ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:34:49.742327Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:34:49.755954Z info tf === SUCCEEDED: Create namespace echo in 13.619403ms === 2025-06-24T10:34:49.779367Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:34:49.784572Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:34:49.784584Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.677184062.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:34:49.784606Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:34:49.784612Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:34:49.788509Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.677184062.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:34:49.788542Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.677184062.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:34:49.788547Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.677184062.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:34:49.788576Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:34:49.788578Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:34:49.797898Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:34:49.797944Z info tf === SUCCEEDED: Cleanup Istio in 40.66µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3445737 2025-06-24T10:35:00.732059Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:35:00.732145Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:35:00.763240Z info tf === Test Framework Settings === 2025-06-24T10:35:00.763326Z info tf TestID: telemetry_tracing_otelcollector RunID: 9b67b006-5994-4b9c-8b47-027b71905a12 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:35:00.763332Z info tf =============================== 2025-06-24T10:35:00.763366Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:35:00.763403Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:35:00.763414Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:35:00.763433Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:35:00.763441Z info tf === BEGIN: Building clusters === 2025-06-24T10:35:00.765900Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:35:00.765934Z info tf === DONE: Building clusters === 2025-06-24T10:35:00.765952Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:35:00.765981Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:35:00.765987Z info tf === Istio Component Config === 2025-06-24T10:35:00.766031Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:35:00.766035Z info tf ================================ 2025-06-24T10:35:00.769319Z info tf skipping deployment as specified in the config 2025-06-24T10:35:00.769332Z info tf === SUCCEEDED: Deploy Istio in 3.347624ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:35:00.769339Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:35:00.786902Z info tf === SUCCEEDED: Create namespace echo in 17.552403ms === 2025-06-24T10:35:00.818350Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:35:00.824123Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:35:00.824143Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3929054162.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:35:00.824178Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:35:00.824272Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:35:00.832533Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3929054162.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:35:00.832590Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3929054162.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:35:00.832596Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3929054162.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:35:00.832619Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:35:00.832617Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:35:00.845200Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:35:00.845278Z info tf === SUCCEEDED: Cleanup Istio in 72.954µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3446960 2025-06-24T10:35:11.490133Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:35:11.490196Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:35:11.518849Z info tf === Test Framework Settings === 2025-06-24T10:35:11.518930Z info tf TestID: telemetry_tracing_otelcollector RunID: 2f232631-14ac-4221-8c72-674b3ee1377f NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:35:11.518936Z info tf =============================== 2025-06-24T10:35:11.518961Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:35:11.519004Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:35:11.519013Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:35:11.519029Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:35:11.519035Z info tf === BEGIN: Building clusters === 2025-06-24T10:35:11.521144Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:35:11.521153Z info tf === DONE: Building clusters === 2025-06-24T10:35:11.521167Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:35:11.521186Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:35:11.521191Z info tf === Istio Component Config === 2025-06-24T10:35:11.521219Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:35:11.521224Z info tf ================================ 2025-06-24T10:35:11.526308Z info tf skipping deployment as specified in the config 2025-06-24T10:35:11.526366Z info tf === SUCCEEDED: Deploy Istio in 5.176933ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:35:11.526373Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:35:11.550774Z info tf === SUCCEEDED: Create namespace echo in 24.39542ms === 2025-06-24T10:35:11.603286Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:35:11.881081Z info klog Waiting for caches to sync for echo 2025-06-24T10:35:11.910418Z info klog Waiting for caches to sync for echo 2025-06-24T10:35:11.983206Z info klog Caches are synced for echo 2025-06-24T10:35:12.010887Z info klog Caches are synced for echo 2025-06-24T10:35:18.287057Z info tf === SUCCEEDED: Deploy echo instances in 6.683769476s === 2025-06-24T10:35:18.329926Z info tf Checking pods ready... 2025-06-24T10:35:18.329966Z info tf Checking pods ready... 2025-06-24T10:35:18.335191Z info tf [ 0] zipkin-77cbd75c85-xllqv Pending (Pending) 2025-06-24T10:35:18.536091Z info tf Checking pods ready... 2025-06-24T10:35:18.536170Z info tf Checking pods ready... 2025-06-24T10:35:18.540993Z info tf [ 0] zipkin-77cbd75c85-xllqv Pending (Pending) 2025-06-24T10:35:18.941389Z info tf Checking pods ready... 2025-06-24T10:35:18.941419Z info tf Checking pods ready... 2025-06-24T10:35:18.949456Z info tf [ 0] zipkin-77cbd75c85-xllqv Pending (Pending) 2025-06-24T10:35:19.749683Z info tf Checking pods ready... 2025-06-24T10:35:19.749722Z info tf Checking pods ready... 2025-06-24T10:35:19.753261Z info tf [ 0] zipkin-77cbd75c85-xllqv Running (container not ready: 'zipkin') 2025-06-24T10:35:21.353939Z info tf Checking pods ready... 2025-06-24T10:35:21.353967Z info tf Checking pods ready... 2025-06-24T10:35:21.358209Z info tf [ 0] zipkin-77cbd75c85-xllqv Running (container not ready: 'zipkin') 2025-06-24T10:35:24.558611Z info tf Checking pods ready... 2025-06-24T10:35:24.558683Z info tf Checking pods ready... 2025-06-24T10:35:24.563338Z info tf [ 0] zipkin-77cbd75c85-xllqv Running (container not ready: 'zipkin') 2025-06-24T10:35:27.763823Z info tf Checking pods ready... 2025-06-24T10:35:27.763873Z info tf Checking pods ready... 2025-06-24T10:35:27.768203Z info tf [ 0] zipkin-77cbd75c85-xllqv Running (container not ready: 'zipkin') 2025-06-24T10:35:30.968854Z info tf Checking pods ready... 2025-06-24T10:35:30.968898Z info tf Checking pods ready... 2025-06-24T10:35:30.972445Z info tf [ 0] zipkin-77cbd75c85-xllqv Running (Ready) 2025-06-24T10:35:31.077294Z info tf Checking pods ready... 2025-06-24T10:35:31.077320Z info tf Checking pods ready... 2025-06-24T10:35:31.082596Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T10:35:31.283205Z info tf Checking pods ready... 2025-06-24T10:35:31.283269Z info tf Checking pods ready... 2025-06-24T10:35:31.288325Z info tf [ 0] opentelemetry-collector-854b84c844-44qw4 Pending (Pending) 2025-06-24T10:35:31.688934Z info tf Checking pods ready... 2025-06-24T10:35:31.688972Z info tf Checking pods ready... 2025-06-24T10:35:31.696770Z info tf [ 0] opentelemetry-collector-854b84c844-44qw4 Pending (Pending) 2025-06-24T10:35:32.497438Z info tf Checking pods ready... 2025-06-24T10:35:32.497488Z info tf Checking pods ready... 2025-06-24T10:35:32.501872Z info tf [ 0] opentelemetry-collector-854b84c844-44qw4 Pending (Pending) 2025-06-24T10:35:34.102382Z info tf Checking pods ready... 2025-06-24T10:35:34.102427Z info tf Checking pods ready... 2025-06-24T10:35:34.106464Z info tf [ 0] opentelemetry-collector-854b84c844-44qw4 Running (Ready) 2025-06-24T10:35:34.149697Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (22.628518866s) === 2025-06-24T10:35:34.149736Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 2025-06-24T10:35:34.162344Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:38:04.162705Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000355161s)' === 2025-06-24T10:38:04.162876Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:38:04.162863Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:38:04.300867Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/2345e056f7-a567-463b-91a9-8f5b195baad4 2025-06-24T10:38:04.301246Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1375s 2025-06-24T10:38:04.301668Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:04.400020Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/239b951696-c17c-4228-8289-184e72a65982 2025-06-24T10:38:04.400093Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2364s 2025-06-24T10:38:04.400733Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:04.509781Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/234c485a29-df6a-436b-a9b6-f73477c8e68e 2025-06-24T10:38:04.509909Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3462s 2025-06-24T10:38:04.510649Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (150.59s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 1) (150.59s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter 2025-06-24T10:35:34.150392Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T10:38:04.749249Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.598852783s)' === 2025-06-24T10:38:04.749347Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:38:04.749350Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:38:04.913542Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/23e80e1b30-b167-4a34-b46c-dc145358b13a 2025-06-24T10:38:04.913663Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1641s 2025-06-24T10:38:04.914341Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.037332Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/239b552278-90db-427c-99e7-2564c573473d 2025-06-24T10:38:05.037440Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2878s 2025-06-24T10:38:05.038123Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.141178Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/231268314c-03a9-4a0d-928c-0627606c693f 2025-06-24T10:38:05.141539Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3919s 2025-06-24T10:38:05.141873Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter (151.15s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 1) (151.15s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T10:35:34.150131Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:36:23.674151Z error port forward failed: lost connection to pod 2025-06-24T10:36:24.043612Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward terminated 2025-06-24T10:36:24.075118Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:36:31.001401Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:37:30.079287Z error port forward failed: lost connection to pod 2025-06-24T10:37:30.544165Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward terminated 2025-06-24T10:37:30.575197Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:38:05.301371Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.151232748s)' === 2025-06-24T10:38:05.301479Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:38:05.301479Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:38:05.434661Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/240bbab46b-d7c1-41cc-9443-95991355aaec 2025-06-24T10:38:05.434966Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1331s 2025-06-24T10:38:05.435374Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.538242Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/247e96b69d-ed41-45a9-b44e-3f33d91af6d5 2025-06-24T10:38:05.538341Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2365s 2025-06-24T10:38:05.538901Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.628215Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/24b107e107-b592-4d58-aefc-255ff2263a6a 2025-06-24T10:38:05.628400Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3265s 2025-06-24T10:38:05.629132Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (151.98s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 1) (151.98s) 2025-06-24T10:38:06.130673Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T10:38:06.130748Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m54.609579317s === 2025-06-24T10:38:06.130769Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T10:38:06.131067Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:38:06.131062Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:38:06.267858Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/245de6ea2c-518a-405f-8236-3b6f7c7014f1 2025-06-24T10:38:06.268133Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1368s 2025-06-24T10:38:06.268758Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:06.356328Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/24867a8ad3-27b4-4b10-a484-b738eb2a3d00 2025-06-24T10:38:06.356432Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2251s 2025-06-24T10:38:06.356922Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:06.458303Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/248743c565-7e60-4aa3-bc7f-d99051c3d81c 2025-06-24T10:38:06.458482Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3271s 2025-06-24T10:38:06.459014Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:06.930682Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:38:06.930749Z info tf === SUCCEEDED: Cleanup Istio in 57.467µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3448499 2025-06-24T10:38:18.160709Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:38:18.160766Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:38:18.192920Z info tf === Test Framework Settings === 2025-06-24T10:38:18.193002Z info tf TestID: telemetry_tracing_otelcollector RunID: bbfd5fa6-2fc1-4449-b5af-b8a7353cf3ca NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:38:18.193008Z info tf =============================== 2025-06-24T10:38:18.193043Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:38:18.193095Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:38:18.193107Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:38:18.193129Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:38:18.193138Z info tf === BEGIN: Building clusters === 2025-06-24T10:38:18.196145Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:38:18.197076Z info tf === DONE: Building clusters === 2025-06-24T10:38:18.197304Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:38:18.197889Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:38:18.197921Z info tf === Istio Component Config === 2025-06-24T10:38:18.197964Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:38:18.197969Z info tf ================================ 2025-06-24T10:38:18.201046Z info tf skipping deployment as specified in the config 2025-06-24T10:38:18.201062Z info tf === SUCCEEDED: Deploy Istio in 3.178317ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:38:18.201068Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:38:18.220854Z info tf === SUCCEEDED: Create namespace echo in 19.775375ms === 2025-06-24T10:38:18.245026Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:38:18.249998Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:38:18.250033Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1893553702.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:38:18.250062Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:38:18.250068Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:38:18.335770Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:38:18.336146Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:38:18.373801Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:13Z/269718c088-6db2-4410-8e9a-0185318c45b5 2025-06-24T10:38:18.373984Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1236s 2025-06-24T10:38:18.374513Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:18.477419Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:13Z/26b0f21690-0bde-45ee-a36d-7f11a42ea2f5 2025-06-24T10:38:18.477565Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2272s 2025-06-24T10:38:18.478015Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:18.523970Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-77c98755c8-xbh5z/istio-proxy for: pods "client-cluster-0-v1-77c98755c8-xbh5z" not found 2025-06-24T10:38:18.525517Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-77c98755c8-xbh5z/istio-validation for: pods "client-cluster-0-v1-77c98755c8-xbh5z" not found 2025-06-24T10:38:18.527539Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-v88s2/app for: pods "server-v1-7d889f5bc4-v88s2" not found 2025-06-24T10:38:18.530982Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-v88s2/istio-proxy for: pods "server-v1-7d889f5bc4-v88s2" not found 2025-06-24T10:38:18.533053Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-v88s2/istio-validation for: pods "server-v1-7d889f5bc4-v88s2" not found 2025-06-24T10:38:18.535042Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1893553702.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:38:18.535051Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1893553702.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:38:18.535057Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1893553702.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:38:18.535078Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:38:18.535114Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:38:18.539895Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:38:18.539910Z info tf === SUCCEEDED: Cleanup Istio in 12.611µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3449711 2025-06-24T10:38:29.464657Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:38:29.464697Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:38:29.495365Z info tf === Test Framework Settings === 2025-06-24T10:38:29.495468Z info tf TestID: telemetry_tracing_otelcollector RunID: e3a0286e-e6b0-4882-a5c8-f959c90790be NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:38:29.495473Z info tf =============================== 2025-06-24T10:38:29.495500Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:38:29.495531Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:38:29.495541Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:38:29.495559Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:38:29.495566Z info tf === BEGIN: Building clusters === 2025-06-24T10:38:29.497767Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:38:29.497774Z info tf === DONE: Building clusters === 2025-06-24T10:38:29.497788Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:38:29.497811Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:38:29.497816Z info tf === Istio Component Config === 2025-06-24T10:38:29.497850Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:38:29.497854Z info tf ================================ 2025-06-24T10:38:29.500846Z info tf skipping deployment as specified in the config 2025-06-24T10:38:29.500859Z info tf === SUCCEEDED: Deploy Istio in 3.04537ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:38:29.500865Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:38:29.524357Z info tf === SUCCEEDED: Create namespace echo in 23.488364ms === 2025-06-24T10:38:29.571428Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:38:29.771459Z info klog Waiting for caches to sync for echo 2025-06-24T10:38:29.782156Z info klog Waiting for caches to sync for echo 2025-06-24T10:38:29.872555Z info klog Caches are synced for echo 2025-06-24T10:38:29.883183Z info klog Caches are synced for echo 2025-06-24T10:38:37.787321Z info tf === SUCCEEDED: Deploy echo instances in 8.215906442s === 2025-06-24T10:38:37.845827Z info tf Checking pods ready... 2025-06-24T10:38:37.845884Z info tf Checking pods ready... 2025-06-24T10:38:37.851425Z info tf [ 0] zipkin-77cbd75c85-mzhhb Pending (Pending) 2025-06-24T10:38:38.052494Z info tf Checking pods ready... 2025-06-24T10:38:38.052526Z info tf Checking pods ready... 2025-06-24T10:38:38.057448Z info tf [ 0] zipkin-77cbd75c85-mzhhb Pending (Pending) 2025-06-24T10:38:38.458219Z info tf Checking pods ready... 2025-06-24T10:38:38.458316Z info tf Checking pods ready... 2025-06-24T10:38:38.462546Z info tf [ 0] zipkin-77cbd75c85-mzhhb Pending (Pending) 2025-06-24T10:38:39.263329Z info tf Checking pods ready... 2025-06-24T10:38:39.263371Z info tf Checking pods ready... 2025-06-24T10:38:39.268044Z info tf [ 0] zipkin-77cbd75c85-mzhhb Pending (Pending) 2025-06-24T10:38:40.868155Z info tf Checking pods ready... 2025-06-24T10:38:40.868195Z info tf Checking pods ready... 2025-06-24T10:38:40.881876Z info tf [ 0] zipkin-77cbd75c85-mzhhb Running (container not ready: 'zipkin') 2025-06-24T10:38:44.082910Z info tf Checking pods ready... 2025-06-24T10:38:44.082957Z info tf Checking pods ready... 2025-06-24T10:38:44.086986Z info tf [ 0] zipkin-77cbd75c85-mzhhb Running (Ready) 2025-06-24T10:38:44.188901Z info tf Checking pods ready... 2025-06-24T10:38:44.188924Z info tf Checking pods ready... 2025-06-24T10:38:44.195263Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T10:38:44.395829Z info tf Checking pods ready... 2025-06-24T10:38:44.395895Z info tf Checking pods ready... 2025-06-24T10:38:44.402868Z info tf [ 0] opentelemetry-collector-854b84c844-nvmx2 Pending (Pending) 2025-06-24T10:38:44.803386Z info tf Checking pods ready... 2025-06-24T10:38:44.803451Z info tf Checking pods ready... 2025-06-24T10:38:44.806802Z info tf [ 0] opentelemetry-collector-854b84c844-nvmx2 Pending (Pending) 2025-06-24T10:38:45.606923Z info tf Checking pods ready... 2025-06-24T10:38:45.606965Z info tf Checking pods ready... 2025-06-24T10:38:45.611178Z info tf [ 0] opentelemetry-collector-854b84c844-nvmx2 Pending (Pending) 2025-06-24T10:38:47.211999Z info tf Checking pods ready... 2025-06-24T10:38:47.212029Z info tf Checking pods ready... 2025-06-24T10:38:47.217022Z info tf [ 0] opentelemetry-collector-854b84c844-nvmx2 Running (Ready) 2025-06-24T10:38:47.255791Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (17.757993029s) === 2025-06-24T10:38:47.255823Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 2025-06-24T10:38:47.264152Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:41:17.265065Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.000907014s)' === 2025-06-24T10:41:17.265156Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:41:17.265145Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:41:17.362511Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/3362c324be-a103-4e97-8564-8c32db4910ce 2025-06-24T10:41:17.363310Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:17.363369Z info tf istioctl ([x internal-debug --all configz]): completed after 0.0978s 2025-06-24T10:41:17.459225Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/3303b247eb-e5d4-4ab1-af69-1c5944fdba66 2025-06-24T10:41:17.459389Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1938s 2025-06-24T10:41:17.460007Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:17.544615Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/33aec11ba3-15e7-4097-9420-849e81bf42e8 2025-06-24T10:41:17.544756Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2792s 2025-06-24T10:41:17.545396Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (150.46s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (re-run 1) (150.46s) === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors 2025-06-24T10:38:47.256240Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T10:41:17.725570Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m30.46932487s)' === 2025-06-24T10:41:17.725639Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:41:17.725656Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:41:17.846621Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/33f32bb676-1f46-4da3-9ea2-cc03c909df2f 2025-06-24T10:41:17.846993Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1212s 2025-06-24T10:41:17.847506Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:17.934788Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/33911bd887-74f3-4e15-bcce-e9a1e28a7d78 2025-06-24T10:41:17.934897Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2091s 2025-06-24T10:41:17.935383Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:18.021524Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/3339a62912-e6c9-431b-8313-6fe301c48ed0 2025-06-24T10:41:18.021702Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2958s 2025-06-24T10:41:18.022211Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors (151.05s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors (re-run 1) (151.05s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T10:38:47.256088Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:39:42.684649Z error port forward failed: lost connection to pod 2025-06-24T10:39:42.959446Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward terminated 2025-06-24T10:39:42.996520Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward reconnect success 2025-06-24T10:39:44.113700Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:40:49.000670Z error port forward failed: lost connection to pod 2025-06-24T10:40:49.459565Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward terminated 2025-06-24T10:40:49.490458Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:41:18.305841Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.049748776s)' === 2025-06-24T10:41:18.305932Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:41:18.305950Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:41:18.426381Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/34c62ba335-8a74-4965-ab7d-386a5dc18795 2025-06-24T10:41:18.426790Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1206s 2025-06-24T10:41:18.427289Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:18.510144Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/3406476c30-02b9-4830-a5e9-b07363946c57 2025-06-24T10:41:18.510403Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2042s 2025-06-24T10:41:18.510954Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:18.603368Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/34e089c52a-b480-460a-9f10-2fb43701041c 2025-06-24T10:41:18.603575Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2974s 2025-06-24T10:41:18.604124Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (151.76s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 1) (151.76s) 2025-06-24T10:41:19.020009Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T10:41:19.020052Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m49.522268049s === 2025-06-24T10:41:19.020064Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T10:41:19.020251Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:41:19.020263Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:41:19.149991Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/3414ca5cd1-4611-44b8-963e-ee36438919fc 2025-06-24T10:41:19.150425Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1299s 2025-06-24T10:41:19.150898Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:19.245323Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/3450de1482-e4cb-4e0d-83eb-5aa793139d7d 2025-06-24T10:41:19.245489Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2249s 2025-06-24T10:41:19.246076Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:19.354927Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/34b90dfff3-72c8-4da7-b89b-f451b8b300d2 2025-06-24T10:41:19.355153Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3346s 2025-06-24T10:41:19.356047Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:19.668585Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:41:19.668729Z info tf === SUCCEEDED: Cleanup Istio in 120.735µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3451247 2025-06-24T10:41:30.493512Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:41:30.493609Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:41:30.534898Z info tf === Test Framework Settings === 2025-06-24T10:41:30.535022Z info tf TestID: telemetry_tracing_otelcollector RunID: 45861a32-b13a-4e20-a633-01721cd0ffad NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:41:30.535029Z info tf =============================== 2025-06-24T10:41:30.535065Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:41:30.535100Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:41:30.535112Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:41:30.535134Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:41:30.535146Z info tf === BEGIN: Building clusters === 2025-06-24T10:41:30.537567Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:41:30.537576Z info tf === DONE: Building clusters === 2025-06-24T10:41:30.537593Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:41:30.537616Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:41:30.537622Z info tf === Istio Component Config === 2025-06-24T10:41:30.537651Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:41:30.537656Z info tf ================================ 2025-06-24T10:41:30.540611Z info tf skipping deployment as specified in the config 2025-06-24T10:41:30.540629Z info tf === SUCCEEDED: Deploy Istio in 3.008459ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:41:30.540635Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:41:30.555981Z info tf === SUCCEEDED: Create namespace echo in 15.335734ms === 2025-06-24T10:41:30.582458Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:41:30.590600Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:41:30.590626Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4065051468.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:41:30.590663Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:41:30.590668Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:41:30.659488Z error klog an error occurred forwarding 40805 -> 15000: error forwarding port 15000 to pod f568f2fc61b182c08b5600bfada8559fc7bd12dca4da6f8aa1fe6995ca4a6366, uid : port forward into network namespace "/var/run/netns/3e039084-6828-4fa9-9162-6227eee661cb": failed to connect to localhost:15000 inside namespace f568f2fc61b182c08b5600bfada8559fc7bd12dca4da6f8aa1fe6995ca4a6366: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:41:30.661469Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-7cnsh for: Get "http://localhost:40805/config_dump?include_eds=true": EOF 2025-06-24T10:41:30.661622Z error port forward failed: lost connection to pod 2025-06-24T10:41:30.662117Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-7cnsh for: Get "http://localhost:40805/clusters": dial tcp 127.0.0.1:40805: connect: connection refused 2025-06-24T10:41:30.668631Z error klog an error occurred forwarding 45909 -> 15020: error forwarding port 15020 to pod 8c246d7faf75de6bd6df1a789e27d8b5a683b1fdbaa8fada67e6205a5ec98b08, uid : port forward into network namespace "/var/run/netns/a772512d-6ed8-4d63-a90b-3817f1cd1b95": failed to connect to localhost:15020 inside namespace 8c246d7faf75de6bd6df1a789e27d8b5a683b1fdbaa8fada67e6205a5ec98b08: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:41:30.670496Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-rcprg for: Get "http://localhost:45909/debug/ndsz": EOF 2025-06-24T10:41:30.670492Z error port forward failed: lost connection to pod 2025-06-24T10:41:30.671106Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-rcprg for: Get "http://localhost:45909/stats/prometheus": dial tcp 127.0.0.1:45909: connect: connection refused 2025-06-24T10:41:30.671903Z error klog an error occurred forwarding 33397 -> 15000: error forwarding port 15000 to pod 8c246d7faf75de6bd6df1a789e27d8b5a683b1fdbaa8fada67e6205a5ec98b08, uid : port forward into network namespace "/var/run/netns/a772512d-6ed8-4d63-a90b-3817f1cd1b95": failed to connect to localhost:15000 inside namespace 8c246d7faf75de6bd6df1a789e27d8b5a683b1fdbaa8fada67e6205a5ec98b08: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:41:30.673636Z error port forward failed: lost connection to pod 2025-06-24T10:41:30.673625Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-rcprg for: Get "http://localhost:33397/config_dump?include_eds=true": EOF 2025-06-24T10:41:30.674435Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-rcprg for: Get "http://localhost:33397/clusters": dial tcp 127.0.0.1:33397: connect: connection refused 2025-06-24T10:41:30.674988Z error klog an error occurred forwarding 41079 -> 15020: error forwarding port 15020 to pod f568f2fc61b182c08b5600bfada8559fc7bd12dca4da6f8aa1fe6995ca4a6366, uid : port forward into network namespace "/var/run/netns/3e039084-6828-4fa9-9162-6227eee661cb": failed to connect to localhost:15020 inside namespace f568f2fc61b182c08b5600bfada8559fc7bd12dca4da6f8aa1fe6995ca4a6366: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:41:30.677357Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-7cnsh for: Get "http://localhost:41079/debug/ndsz": EOF 2025-06-24T10:41:30.677411Z error port forward failed: lost connection to pod 2025-06-24T10:41:30.677933Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-7cnsh for: Get "http://localhost:41079/stats/prometheus": dial tcp 127.0.0.1:41079: connect: connection refused 2025-06-24T10:41:30.709086Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:25Z/367b850331-da6f-4214-99a8-7745a5b742e4 2025-06-24T10:41:30.709312Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1183s 2025-06-24T10:41:30.709974Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:30.819158Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:25Z/363736ce28-19d7-47df-baf6-7db54cd6e304 2025-06-24T10:41:30.819376Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2283s 2025-06-24T10:41:30.820067Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:30.925359Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:25Z/36b6314d69-338f-428d-b7bc-126d87f75454 2025-06-24T10:41:30.925551Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3344s 2025-06-24T10:41:30.926183Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:30.931859Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4065051468.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:41:30.931909Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4065051468.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:41:30.931918Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4065051468.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:41:30.931950Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:41:30.931969Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:41:30.974735Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7d889f5bc4-7cnsh_echo") 2025-06-24T10:41:30.975246Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-rcprg_echo") 2025-06-24T10:41:30.980132Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-rcprg_echo") 2025-06-24T10:41:30.980176Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7d889f5bc4-7cnsh_echo") 2025-06-24T10:41:31.047379Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7d889f5bc4-7cnsh_echo") 2025-06-24T10:41:31.051118Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-rcprg_echo") 2025-06-24T10:41:31.053211Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7d889f5bc4-7cnsh_echo") 2025-06-24T10:41:31.056852Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-rcprg_echo") 2025-06-24T10:41:31.069442Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:25Z/36c2651efe-a2af-43b7-833f-4e126c590aa2 2025-06-24T10:41:31.069704Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1376s 2025-06-24T10:41:31.070180Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:31.106132Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-rcprg_echo") 2025-06-24T10:41:31.127743Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-rcprg_echo") 2025-06-24T10:41:31.141819Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:41:31.164586Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:41:31.176288Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:25Z/3643e75603-0c75-4fe0-8736-7dce8b113147 2025-06-24T10:41:31.176518Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2444s 2025-06-24T10:41:31.177005Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:31.267765Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:25Z/36303941d3-8c9b-46ce-8b41-ba60cf15e00b 2025-06-24T10:41:31.267991Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3358s 2025-06-24T10:41:31.268447Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:31.287594Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:41:31.287667Z info tf === SUCCEEDED: Cleanup Istio in 66.579µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3452524 2025-06-24T10:41:41.795828Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:41:41.795897Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:41:41.825791Z info tf === Test Framework Settings === 2025-06-24T10:41:41.825873Z info tf TestID: telemetry_tracing_otelcollector RunID: 16c17cc8-389e-44bd-a028-3ebca94d0c89 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:41:41.825879Z info tf =============================== 2025-06-24T10:41:41.825905Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:41:41.825936Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:41:41.825945Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:41:41.825960Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:41:41.825967Z info tf === BEGIN: Building clusters === 2025-06-24T10:41:41.828081Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:41:41.828089Z info tf === DONE: Building clusters === 2025-06-24T10:41:41.828104Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:41:41.828123Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:41:41.828128Z info tf === Istio Component Config === 2025-06-24T10:41:41.828156Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:41:41.828160Z info tf ================================ 2025-06-24T10:41:41.831015Z info tf skipping deployment as specified in the config 2025-06-24T10:41:41.831029Z info tf === SUCCEEDED: Deploy Istio in 2.903548ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:41:41.831036Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:41:41.858554Z info tf === SUCCEEDED: Create namespace echo in 27.502148ms === 2025-06-24T10:41:41.956648Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:41:42.241183Z info klog Waiting for caches to sync for echo 2025-06-24T10:41:42.241480Z info klog Waiting for caches to sync for echo 2025-06-24T10:41:42.342346Z info klog Caches are synced for echo 2025-06-24T10:41:42.342381Z info klog Caches are synced for echo 2025-06-24T10:41:48.660180Z info tf === SUCCEEDED: Deploy echo instances in 6.703530688s === 2025-06-24T10:41:48.719852Z info tf Checking pods ready... 2025-06-24T10:41:48.719897Z info tf Checking pods ready... 2025-06-24T10:41:48.727502Z info tf [ 0] zipkin-77cbd75c85-2r2s8 Pending (Pending) 2025-06-24T10:41:48.927690Z info tf Checking pods ready... 2025-06-24T10:41:48.927758Z info tf Checking pods ready... 2025-06-24T10:41:48.932230Z info tf [ 0] zipkin-77cbd75c85-2r2s8 Pending (Pending) 2025-06-24T10:41:49.333214Z info tf Checking pods ready... 2025-06-24T10:41:49.333295Z info tf Checking pods ready... 2025-06-24T10:41:49.338852Z info tf [ 0] zipkin-77cbd75c85-2r2s8 Pending (Pending) 2025-06-24T10:41:50.139674Z info tf Checking pods ready... 2025-06-24T10:41:50.139716Z info tf Checking pods ready... 2025-06-24T10:41:50.143858Z info tf [ 0] zipkin-77cbd75c85-2r2s8 Pending (Pending) 2025-06-24T10:41:51.744167Z info tf Checking pods ready... 2025-06-24T10:41:51.744215Z info tf Checking pods ready... 2025-06-24T10:41:51.747992Z info tf [ 0] zipkin-77cbd75c85-2r2s8 Running (container not ready: 'zipkin') 2025-06-24T10:41:54.948588Z info tf Checking pods ready... 2025-06-24T10:41:54.948662Z info tf Checking pods ready... 2025-06-24T10:41:54.953846Z info tf [ 0] zipkin-77cbd75c85-2r2s8 Running (Ready) 2025-06-24T10:41:55.103157Z info tf Checking pods ready... 2025-06-24T10:41:55.103185Z info tf Checking pods ready... 2025-06-24T10:41:55.106713Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T10:41:55.307348Z info tf Checking pods ready... 2025-06-24T10:41:55.307426Z info tf Checking pods ready... 2025-06-24T10:41:55.313058Z info tf [ 0] opentelemetry-collector-854b84c844-bjp7d Pending (Pending) 2025-06-24T10:41:55.713412Z info tf Checking pods ready... 2025-06-24T10:41:55.713444Z info tf Checking pods ready... 2025-06-24T10:41:55.725040Z info tf [ 0] opentelemetry-collector-854b84c844-bjp7d Pending (Pending) 2025-06-24T10:41:56.525518Z info tf Checking pods ready... 2025-06-24T10:41:56.525584Z info tf Checking pods ready... 2025-06-24T10:41:56.530084Z info tf [ 0] opentelemetry-collector-854b84c844-bjp7d Pending (Pending) 2025-06-24T10:41:58.131017Z info tf Checking pods ready... 2025-06-24T10:41:58.131066Z info tf Checking pods ready... 2025-06-24T10:41:58.135195Z info tf [ 0] opentelemetry-collector-854b84c844-bjp7d Running (Ready) 2025-06-24T10:41:58.188200Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (16.360083851s) === 2025-06-24T10:41:58.188243Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 2025-06-24T10:41:58.196552Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:44:28.196941Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000379685s)' === 2025-06-24T10:44:28.197058Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:44:28.197060Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:44:28.312919Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/44b7454c35-080a-4aab-88bc-0589eb888154 2025-06-24T10:44:28.313055Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1156s 2025-06-24T10:44:28.313635Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.405816Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/44becb45cf-690a-41e4-9cbb-88c7763ed25f 2025-06-24T10:44:28.406104Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2086s 2025-06-24T10:44:28.406581Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.498958Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/441fbd5eb0-4a30-4538-97ef-3fee68bb40d3 2025-06-24T10:44:28.499098Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3016s 2025-06-24T10:44:28.499735Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (150.45s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 1) (150.45s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata 2025-06-24T10:41:58.188771Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T10:44:28.643229Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.454454386s)' === 2025-06-24T10:44:28.643311Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:44:28.643322Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:44:28.760454Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/44018f0064-0043-4f0a-8f4a-02bb18d244a3 2025-06-24T10:44:28.760737Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1173s 2025-06-24T10:44:28.761221Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.851166Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/447beeb700-0d89-46e1-b4de-023c07fe7696 2025-06-24T10:44:28.851368Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2079s 2025-06-24T10:44:28.851984Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.936515Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/447ff678fa-9308-4a53-90ee-5e00bfa5e283 2025-06-24T10:44:28.936666Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2932s 2025-06-24T10:44:28.937142Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (150.97s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 1) (150.97s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T10:41:58.188542Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:42:54.664830Z error port forward failed: lost connection to pod 2025-06-24T10:42:54.840355Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward terminated 2025-06-24T10:42:54.876821Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward reconnect success 2025-06-24T10:42:54.999665Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:44:00.881841Z error port forward failed: lost connection to pod 2025-06-24T10:44:01.339993Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward terminated 2025-06-24T10:44:01.381850Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:44:29.157047Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m30.968502163s)' === 2025-06-24T10:44:29.157111Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:44:29.157110Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:44:29.280802Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/4534137018-3643-4ebb-ba87-bb23a484224b 2025-06-24T10:44:29.281103Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1237s 2025-06-24T10:44:29.281647Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:29.361759Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/4575c6fedb-173e-4f82-9657-cc2f97d64eb7 2025-06-24T10:44:29.361927Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2046s 2025-06-24T10:44:29.362477Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:29.450245Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/4593de1097-44f8-420e-af7e-d29729694b97 2025-06-24T10:44:29.450512Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2931s 2025-06-24T10:44:29.450966Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (151.68s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 1) (151.68s) 2025-06-24T10:44:29.872032Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T10:44:29.872104Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m48.044003273s === 2025-06-24T10:44:29.872119Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T10:44:29.872345Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:29.872341Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:29.969954Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/45ade48cbb-461e-4cd0-b07a-7ca1594af073 2025-06-24T10:44:29.970104Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.0974s 2025-06-24T10:44:29.970784Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:30.068845Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/45b8f6fca1-623e-453c-abf2-99a449dfd71a 2025-06-24T10:44:30.069161Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1964s 2025-06-24T10:44:30.069595Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:30.173358Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/45a2a5a559-1c1a-46c6-bd1c-912a844c0a71 2025-06-24T10:44:30.173480Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3008s 2025-06-24T10:44:30.174118Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:30.473142Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:44:30.473223Z info tf === SUCCEEDED: Cleanup Istio in 68.839µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3454077 2025-06-24T10:44:40.931853Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:44:40.931919Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:44:40.962917Z info tf === Test Framework Settings === 2025-06-24T10:44:40.963040Z info tf TestID: telemetry_tracing_otelcollector RunID: f902f275-de39-4574-bb5e-37f2a7b6318b NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:44:40.963047Z info tf =============================== 2025-06-24T10:44:40.963078Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:44:40.963122Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:44:40.963134Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:44:40.963152Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:44:40.963161Z info tf === BEGIN: Building clusters === 2025-06-24T10:44:40.965984Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:44:40.965992Z info tf === DONE: Building clusters === 2025-06-24T10:44:40.966008Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:44:40.966026Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:44:40.966031Z info tf === Istio Component Config === 2025-06-24T10:44:40.966057Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:44:40.966061Z info tf ================================ 2025-06-24T10:44:40.969334Z info tf skipping deployment as specified in the config 2025-06-24T10:44:40.969351Z info tf === SUCCEEDED: Deploy Istio in 3.320401ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:44:40.969360Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:44:40.992502Z info tf === SUCCEEDED: Create namespace echo in 23.131056ms === 2025-06-24T10:44:41.013640Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:44:41.019531Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:44:41.019553Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2609737958.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:44:41.019582Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:41.019591Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:41.100919Z error klog an error occurred forwarding 44735 -> 15020: error forwarding port 15020 to pod 008f6a0629877d9d3f204177825df52e3966d123269f74627ee4ec242b540a53, uid : port forward into network namespace "/var/run/netns/a542627a-b58c-4330-b820-93fb735eccc8": failed to connect to localhost:15020 inside namespace 008f6a0629877d9d3f204177825df52e3966d123269f74627ee4ec242b540a53: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:44:41.102538Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-q67dg for: Get "http://localhost:44735/debug/ndsz": EOF 2025-06-24T10:44:41.102538Z error port forward failed: lost connection to pod 2025-06-24T10:44:41.103334Z error klog an error occurred forwarding 39729 -> 15020: error forwarding port 15020 to pod 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71, uid : port forward into network namespace "/var/run/netns/e26244c0-e11c-423f-9329-8c9fb41bffca": failed to connect to localhost:15020 inside namespace 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:44:41.104210Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-q67dg for: Get "http://localhost:44735/stats/prometheus": dial tcp 127.0.0.1:44735: connect: connection refused 2025-06-24T10:44:41.104762Z error port forward failed: lost connection to pod 2025-06-24T10:44:41.104784Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-bmfm9 for: Get "http://localhost:39729/debug/ndsz": EOF 2025-06-24T10:44:41.105346Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-bmfm9 for: Get "http://localhost:39729/stats/prometheus": dial tcp 127.0.0.1:39729: connect: connection refused 2025-06-24T10:44:41.123688Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:36Z/47a27b28b9-95ee-4dae-9ce4-ed40bb770d32 2025-06-24T10:44:41.123852Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1041s 2025-06-24T10:44:41.124348Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:41.208712Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:36Z/47ce04fbac-4cda-4e89-96af-515073b80490 2025-06-24T10:44:41.208800Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.1889s 2025-06-24T10:44:41.209297Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:41.306221Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:36Z/47575eb331-39ea-4b58-86ec-5da9dfe052d3 2025-06-24T10:44:41.306357Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2865s 2025-06-24T10:44:41.306910Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:41.553363Z error klog an error occurred forwarding 45161 -> 15000: error forwarding port 15000 to pod 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71, uid : port forward into network namespace "/var/run/netns/e26244c0-e11c-423f-9329-8c9fb41bffca": failed to connect to localhost:15000 inside namespace 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:44:41.555504Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-bmfm9 for: Get "http://localhost:45161/clusters": EOF 2025-06-24T10:44:41.555563Z error port forward failed: lost connection to pod 2025-06-24T10:44:41.559565Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2609737958.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:44:41.559582Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2609737958.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:44:41.559587Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2609737958.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:44:41.559608Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:41.559606Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:41.618092Z error klog an error occurred forwarding 45999 -> 15000: error forwarding port 15000 to pod 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71, uid : port forward into network namespace "/var/run/netns/e26244c0-e11c-423f-9329-8c9fb41bffca": failed to connect to localhost:15000 inside namespace 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:44:41.619785Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-bmfm9 for: Get "http://localhost:45999/config_dump?include_eds=true": EOF 2025-06-24T10:44:41.619825Z error port forward failed: lost connection to pod 2025-06-24T10:44:41.620655Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-bmfm9 for: Get "http://localhost:45999/clusters": dial tcp 127.0.0.1:45999: connect: connection refused 2025-06-24T10:44:41.621578Z error klog an error occurred forwarding 34125 -> 15020: error forwarding port 15020 to pod 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71, uid : port forward into network namespace "/var/run/netns/e26244c0-e11c-423f-9329-8c9fb41bffca": failed to connect to localhost:15020 inside namespace 018e9be8b52c5d77bbf8909354796ff017eaf9953187f91b8704d42d4990dc71: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:44:41.623141Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-bmfm9 for: Get "http://localhost:34125/debug/ndsz": EOF 2025-06-24T10:44:41.623164Z error port forward failed: lost connection to pod 2025-06-24T10:44:41.623459Z error klog an error occurred forwarding 37753 -> 15000: error forwarding port 15000 to pod 008f6a0629877d9d3f204177825df52e3966d123269f74627ee4ec242b540a53, uid : port forward into network namespace "/var/run/netns/a542627a-b58c-4330-b820-93fb735eccc8": failed to connect to localhost:15000 inside namespace 008f6a0629877d9d3f204177825df52e3966d123269f74627ee4ec242b540a53: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:44:41.623646Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-bmfm9 for: Get "http://localhost:34125/stats/prometheus": dial tcp 127.0.0.1:34125: connect: connection refused 2025-06-24T10:44:41.624969Z error port forward failed: lost connection to pod 2025-06-24T10:44:41.624954Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-q67dg for: Get "http://localhost:37753/config_dump?include_eds=true": EOF 2025-06-24T10:44:41.625489Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-q67dg for: Get "http://localhost:37753/clusters": dial tcp 127.0.0.1:37753: connect: connection refused 2025-06-24T10:44:41.626286Z error klog an error occurred forwarding 43361 -> 15020: error forwarding port 15020 to pod 008f6a0629877d9d3f204177825df52e3966d123269f74627ee4ec242b540a53, uid : port forward into network namespace "/var/run/netns/a542627a-b58c-4330-b820-93fb735eccc8": failed to connect to localhost:15020 inside namespace 008f6a0629877d9d3f204177825df52e3966d123269f74627ee4ec242b540a53: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:44:41.627913Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-q67dg for: Get "http://localhost:43361/debug/ndsz": EOF 2025-06-24T10:44:41.628058Z error port forward failed: lost connection to pod 2025-06-24T10:44:41.628426Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-q67dg for: Get "http://localhost:43361/stats/prometheus": dial tcp 127.0.0.1:43361: connect: connection refused 2025-06-24T10:44:41.709081Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:36Z/476642a941-51e7-4735-8f03-09eccca9737c 2025-06-24T10:44:41.709223Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1494s 2025-06-24T10:44:41.709796Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:41.840323Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:36Z/478d556222-3c4c-4f87-89d8-10682f10d45c 2025-06-24T10:44:41.840432Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2805s 2025-06-24T10:44:41.841046Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:41.901999Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:44:41.902046Z info tf === SUCCEEDED: Cleanup Istio in 40.727µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector DONE 2 runs, 18 tests, 18 failures in 1301.699s exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3455450 2025-06-24T10:44:52.818276Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:44:52.818322Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:44:52.851453Z info tf === Test Framework Settings === 2025-06-24T10:44:52.851542Z info tf TestID: telemetry_tracing_otelcollector RunID: 5e7a6d4e-b53d-41a3-8be7-17c96ddefafd NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:44:52.851547Z info tf =============================== 2025-06-24T10:44:52.851695Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:44:52.851741Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:44:52.851752Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:44:52.851768Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:44:52.851776Z info tf === BEGIN: Building clusters === 2025-06-24T10:44:52.856410Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:44:52.856477Z info tf === DONE: Building clusters === 2025-06-24T10:44:52.856508Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:44:52.856535Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:44:52.856542Z info tf === Istio Component Config === 2025-06-24T10:44:52.856581Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:44:52.856586Z info tf ================================ 2025-06-24T10:44:52.859906Z info tf skipping deployment as specified in the config 2025-06-24T10:44:52.859923Z info tf === SUCCEEDED: Deploy Istio in 3.382412ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:44:52.859930Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:44:52.874184Z info tf === SUCCEEDED: Create namespace echo in 14.241667ms === 2025-06-24T10:44:52.898605Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:44:52.909318Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:44:52.909362Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3366124471.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:44:52.909398Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:52.909427Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:52.913748Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3366124471.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:44:52.913761Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3366124471.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:44:52.913766Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3366124471.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:44:52.913785Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:52.913842Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:44:52.922369Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:44:52.922455Z info tf === SUCCEEDED: Cleanup Istio in 80.487µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3456794 2025-06-24T10:45:03.502318Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:45:03.502367Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:45:03.545666Z info tf === Test Framework Settings === 2025-06-24T10:45:03.545769Z info tf TestID: telemetry_tracing_otelcollector RunID: 88b81cfc-04c3-4c49-b7cb-ab4c3103cc2b NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:45:03.545778Z info tf =============================== 2025-06-24T10:45:03.545823Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:45:03.545871Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:45:03.545888Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:45:03.545911Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:45:03.545924Z info tf === BEGIN: Building clusters === 2025-06-24T10:45:03.548986Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:45:03.548995Z info tf === DONE: Building clusters === 2025-06-24T10:45:03.549010Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:45:03.549032Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:45:03.549037Z info tf === Istio Component Config === 2025-06-24T10:45:03.549068Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:45:03.549076Z info tf ================================ 2025-06-24T10:45:03.553531Z info tf skipping deployment as specified in the config 2025-06-24T10:45:03.553549Z info tf === SUCCEEDED: Deploy Istio in 4.512468ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:45:03.553556Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:45:03.578043Z info tf === SUCCEEDED: Create namespace echo in 24.482365ms === 2025-06-24T10:45:03.631056Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:45:04.137504Z info klog Waiting for caches to sync for echo 2025-06-24T10:45:04.174260Z info klog Waiting for caches to sync for echo 2025-06-24T10:45:04.238619Z info klog Caches are synced for echo 2025-06-24T10:45:04.275269Z info klog Caches are synced for echo 2025-06-24T10:45:12.180744Z info tf === SUCCEEDED: Deploy echo instances in 8.549663478s === 2025-06-24T10:45:12.223730Z info tf Checking pods ready... 2025-06-24T10:45:12.223749Z info tf Checking pods ready... 2025-06-24T10:45:12.234285Z info tf [ 0] zipkin-77cbd75c85-lkd5g Pending (Pending) 2025-06-24T10:45:12.435389Z info tf Checking pods ready... 2025-06-24T10:45:12.435424Z info tf Checking pods ready... 2025-06-24T10:45:12.439127Z info tf [ 0] zipkin-77cbd75c85-lkd5g Pending (Pending) 2025-06-24T10:45:12.839543Z info tf Checking pods ready... 2025-06-24T10:45:12.839621Z info tf Checking pods ready... 2025-06-24T10:45:12.843328Z info tf [ 0] zipkin-77cbd75c85-lkd5g Pending (Pending) 2025-06-24T10:45:13.644457Z info tf Checking pods ready... 2025-06-24T10:45:13.644529Z info tf Checking pods ready... 2025-06-24T10:45:13.647860Z info tf [ 0] zipkin-77cbd75c85-lkd5g Pending (Pending) 2025-06-24T10:45:15.248507Z info tf Checking pods ready... 2025-06-24T10:45:15.248589Z info tf Checking pods ready... 2025-06-24T10:45:15.252284Z info tf [ 0] zipkin-77cbd75c85-lkd5g Running (container not ready: 'zipkin') 2025-06-24T10:45:18.453197Z info tf Checking pods ready... 2025-06-24T10:45:18.453246Z info tf Checking pods ready... 2025-06-24T10:45:18.458963Z info tf [ 0] zipkin-77cbd75c85-lkd5g Running (container not ready: 'zipkin') 2025-06-24T10:45:21.660882Z info tf Checking pods ready... 2025-06-24T10:45:21.660982Z info tf Checking pods ready... 2025-06-24T10:45:21.667840Z info tf [ 0] zipkin-77cbd75c85-lkd5g Running (container not ready: 'zipkin') 2025-06-24T10:45:24.868216Z info tf Checking pods ready... 2025-06-24T10:45:24.868291Z info tf Checking pods ready... 2025-06-24T10:45:24.872269Z info tf [ 0] zipkin-77cbd75c85-lkd5g Running (Ready) 2025-06-24T10:45:24.966680Z info tf Checking pods ready... 2025-06-24T10:45:24.966717Z info tf Checking pods ready... 2025-06-24T10:45:24.970055Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T10:45:25.170649Z info tf Checking pods ready... 2025-06-24T10:45:25.170701Z info tf Checking pods ready... 2025-06-24T10:45:25.175150Z info tf [ 0] opentelemetry-collector-854b84c844-n5ctj Pending (Pending) 2025-06-24T10:45:25.576271Z info tf Checking pods ready... 2025-06-24T10:45:25.576305Z info tf Checking pods ready... 2025-06-24T10:45:25.580945Z info tf [ 0] opentelemetry-collector-854b84c844-n5ctj Pending (Pending) 2025-06-24T10:45:26.380976Z info tf Checking pods ready... 2025-06-24T10:45:26.381012Z info tf Checking pods ready... 2025-06-24T10:45:26.385092Z info tf [ 0] opentelemetry-collector-854b84c844-n5ctj Pending (Pending) 2025-06-24T10:45:27.985327Z info tf Checking pods ready... 2025-06-24T10:45:27.985418Z info tf Checking pods ready... 2025-06-24T10:45:27.990686Z info tf [ 0] opentelemetry-collector-854b84c844-n5ctj Running (Ready) 2025-06-24T10:45:28.030651Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (24.481628703s) === 2025-06-24T10:45:28.030686Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 2025-06-24T10:45:28.039767Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:47:58.040488Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000717748s)' === 2025-06-24T10:47:58.040588Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:47:58.040588Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:47:58.159384Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/555ada0cec-48ec-45b3-88cc-3a31a7e63ae6 2025-06-24T10:47:58.159578Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1184s 2025-06-24T10:47:58.160197Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.246024Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/5579f7f84d-dcbe-423b-830a-9e06f59e5fae 2025-06-24T10:47:58.246145Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2049s 2025-06-24T10:47:58.246747Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.385060Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/55c733a5b3-a3f0-400d-a4a2-8483997cdf0e 2025-06-24T10:47:58.385466Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3443s 2025-06-24T10:47:58.385974Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (150.45s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 2) (150.45s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter 2025-06-24T10:45:28.031126Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T10:47:58.491625Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.460496547s)' === 2025-06-24T10:47:58.491687Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:47:58.491689Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:47:58.595966Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/553c7c52a8-924e-408d-a876-9c3ad1e303b9 2025-06-24T10:47:58.596232Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1044s 2025-06-24T10:47:58.596649Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.684805Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/55dc7ee313-3ec1-4d69-9f3a-c00f4982d35f 2025-06-24T10:47:58.684973Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1931s 2025-06-24T10:47:58.685575Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.770739Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/55c9e8d6f5-e7e6-4252-8f89-3b3996a5ff94 2025-06-24T10:47:58.770878Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2790s 2025-06-24T10:47:58.771355Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter (150.94s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 2) (150.94s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T10:45:28.030968Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:46:16.801049Z error port forward failed: lost connection to pod 2025-06-24T10:46:16.826567Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward terminated 2025-06-24T10:46:16.860829Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:46:24.897306Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:47:22.864512Z error port forward failed: lost connection to pod 2025-06-24T10:47:23.326351Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward terminated 2025-06-24T10:47:23.359151Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:47:58.975232Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m30.944257413s)' === 2025-06-24T10:47:58.975337Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:47:58.975330Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:47:59.084698Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/561c702580-a6fc-40b6-96e1-1652545ca728 2025-06-24T10:47:59.085134Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1096s 2025-06-24T10:47:59.085543Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:59.191186Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/564fe66857-201a-4958-b828-26a12c6d7959 2025-06-24T10:47:59.191343Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2157s 2025-06-24T10:47:59.191961Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:59.289805Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/56505d8648-48b7-472b-95ec-d5b6a7793c5f 2025-06-24T10:47:59.290201Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3146s 2025-06-24T10:47:59.290799Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (151.59s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 2) (151.59s) 2025-06-24T10:47:59.625431Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T10:47:59.625492Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m56.076483084s === 2025-06-24T10:47:59.625507Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T10:47:59.625726Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:47:59.625725Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:47:59.744699Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/561761a2c6-d4cc-4236-a6ab-4fbbf662a113 2025-06-24T10:47:59.744992Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1190s 2025-06-24T10:47:59.745433Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:59.881621Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/568a9f61b6-f00c-4985-9a8e-a9250dda1b24 2025-06-24T10:47:59.881779Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2558s 2025-06-24T10:47:59.882344Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:00.123805Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/56b7472332-8e0c-4754-86f2-7ab6e6f1f35d 2025-06-24T10:48:00.123982Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4980s 2025-06-24T10:48:00.124618Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:00.469740Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:48:00.469846Z info tf === SUCCEEDED: Cleanup Istio in 96.657µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3458299 2025-06-24T10:48:11.134189Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:48:11.134243Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:48:11.163999Z info tf === Test Framework Settings === 2025-06-24T10:48:11.164095Z info tf TestID: telemetry_tracing_otelcollector RunID: 43e21bb0-074e-4f71-bc82-9da6215be364 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:48:11.164100Z info tf =============================== 2025-06-24T10:48:11.164291Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:48:11.164334Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:48:11.164343Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:48:11.164359Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:48:11.164366Z info tf === BEGIN: Building clusters === 2025-06-24T10:48:11.168970Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:48:11.169024Z info tf === DONE: Building clusters === 2025-06-24T10:48:11.169052Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:48:11.169091Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:48:11.169099Z info tf === Istio Component Config === 2025-06-24T10:48:11.169149Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:48:11.169156Z info tf ================================ 2025-06-24T10:48:11.172474Z info tf skipping deployment as specified in the config 2025-06-24T10:48:11.172494Z info tf === SUCCEEDED: Deploy Istio in 3.398959ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:48:11.172503Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:48:11.186257Z info tf === SUCCEEDED: Create namespace echo in 13.735072ms === 2025-06-24T10:48:11.213418Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:48:11.224060Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:48:11.224080Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.259188875.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:48:11.224117Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:11.224131Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:11.427207Z error klog an error occurred forwarding 46075 -> 15020: error forwarding port 15020 to pod 656e9e22c28a7dfd85de8d2e2ce308d62e52b66fedfc0441bcddc76edeae7cf8, uid : port forward into network namespace "/var/run/netns/65511fc0-0714-4c30-8187-a65305bf0248": failed to connect to localhost:15020 inside namespace 656e9e22c28a7dfd85de8d2e2ce308d62e52b66fedfc0441bcddc76edeae7cf8: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:48:11.429487Z error port forward failed: lost connection to pod 2025-06-24T10:48:11.429505Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-kt8xq for: Get "http://localhost:46075/debug/ndsz": EOF 2025-06-24T10:48:11.430488Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-kt8xq for: Get "http://localhost:46075/stats/prometheus": dial tcp 127.0.0.1:46075: connect: connection refused 2025-06-24T10:48:11.512346Z error klog an error occurred forwarding 34629 -> 15020: error forwarding port 15020 to pod f5c962b870b97c99930594f47c4e34a278f709b35afa68821bd391d755fb7289, uid : port forward into network namespace "/var/run/netns/83cf8e0e-051d-404e-a6cf-e86a3c8fa846": failed to connect to localhost:15020 inside namespace f5c962b870b97c99930594f47c4e34a278f709b35afa68821bd391d755fb7289: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:48:11.514275Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-6l4jj for: Get "http://localhost:34629/debug/ndsz": EOF 2025-06-24T10:48:11.514289Z error port forward failed: lost connection to pod 2025-06-24T10:48:11.514891Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-6l4jj for: Get "http://localhost:34629/stats/prometheus": dial tcp 127.0.0.1:34629: connect: connection refused 2025-06-24T10:48:11.548032Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:06Z/580c139f25-ef6a-46ed-89fa-50a12d700123 2025-06-24T10:48:11.548183Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3238s 2025-06-24T10:48:11.548704Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:11.628568Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:06Z/5847928ac7-7721-47a9-a324-5994789907d4 2025-06-24T10:48:11.628651Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4042s 2025-06-24T10:48:11.629094Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:11.717608Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:06Z/58e7fba7c5-b6cb-49fe-9309-d8ff2c7ab4dd 2025-06-24T10:48:11.717749Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4933s 2025-06-24T10:48:11.718162Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:12.515035Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-6l4jj for: Get "http://localhost:40247/config_dump?include_eds=true": EOF 2025-06-24T10:48:12.518534Z error klog an error occurred forwarding 40247 -> 15000: error forwarding port 15000 to pod f5c962b870b97c99930594f47c4e34a278f709b35afa68821bd391d755fb7289, uid : sandbox f5c962b870b97c99930594f47c4e34a278f709b35afa68821bd391d755fb7289 is not running: Unhandled Error logger=UnhandledError 2025-06-24T10:48:12.520191Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-6l4jj for: Get "http://localhost:40247/clusters": EOF 2025-06-24T10:48:12.520215Z error port forward failed: lost connection to pod 2025-06-24T10:48:12.536021Z error klog an error occurred forwarding 36119 -> 15000: error forwarding port 15000 to pod 656e9e22c28a7dfd85de8d2e2ce308d62e52b66fedfc0441bcddc76edeae7cf8, uid : port forward into network namespace "/var/run/netns/65511fc0-0714-4c30-8187-a65305bf0248": read tcp 127.0.0.1:35486->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:48:12.538020Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-kt8xq for: Get "http://localhost:36119/config_dump?include_eds=true": EOF 2025-06-24T10:48:12.538036Z error port forward failed: lost connection to pod 2025-06-24T10:48:12.538642Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-kt8xq for: Get "http://localhost:36119/clusters": dial tcp 127.0.0.1:36119: connect: connection refused 2025-06-24T10:48:12.541125Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.259188875.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:48:12.541135Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.259188875.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:48:12.541140Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.259188875.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:48:12.541158Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:12.541160Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:12.630347Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:48:12.630417Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:48:12.671734Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:06Z/58731310c3-78ae-48f3-a662-da94bae284a1 2025-06-24T10:48:12.671925Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1305s 2025-06-24T10:48:12.672535Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:12.752895Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:06Z/58228a20f9-f22d-4944-8d37-0be3178b47aa 2025-06-24T10:48:12.752998Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2116s 2025-06-24T10:48:12.753516Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:12.837397Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:06Z/5873ecea6f-1ad7-4be0-b667-ada75d107166 2025-06-24T10:48:12.837591Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2962s 2025-06-24T10:48:12.838027Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:48:12.847840Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-77c98755c8-kt8xq/istio-proxy for: pods "client-cluster-0-v1-77c98755c8-kt8xq" not found 2025-06-24T10:48:12.850210Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-77c98755c8-kt8xq/istio-validation for: pods "client-cluster-0-v1-77c98755c8-kt8xq" not found 2025-06-24T10:48:12.855787Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-6l4jj/app for: pods "server-v1-7d889f5bc4-6l4jj" not found 2025-06-24T10:48:12.861062Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-6l4jj/istio-proxy for: pods "server-v1-7d889f5bc4-6l4jj" not found 2025-06-24T10:48:12.863078Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-7d889f5bc4-6l4jj/istio-validation for: pods "server-v1-7d889f5bc4-6l4jj" not found 2025-06-24T10:48:12.865676Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:48:12.865717Z info tf === SUCCEEDED: Cleanup Istio in 34.325µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3459604 2025-06-24T10:48:24.016826Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:48:24.016876Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:48:24.047350Z info tf === Test Framework Settings === 2025-06-24T10:48:24.047416Z info tf TestID: telemetry_tracing_otelcollector RunID: a9f960c3-d991-403f-ae27-17a4febc8829 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:48:24.047422Z info tf =============================== 2025-06-24T10:48:24.047448Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:48:24.047481Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:48:24.047490Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:48:24.047507Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:48:24.047513Z info tf === BEGIN: Building clusters === 2025-06-24T10:48:24.049660Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:48:24.049670Z info tf === DONE: Building clusters === 2025-06-24T10:48:24.049682Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:48:24.049706Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:48:24.049711Z info tf === Istio Component Config === 2025-06-24T10:48:24.049737Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:48:24.049741Z info tf ================================ 2025-06-24T10:48:24.052570Z info tf skipping deployment as specified in the config 2025-06-24T10:48:24.052583Z info tf === SUCCEEDED: Deploy Istio in 2.874088ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:48:24.052589Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:48:24.067177Z info tf === SUCCEEDED: Create namespace echo in 14.572872ms === 2025-06-24T10:48:24.091561Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:48:24.098721Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:48:24.098736Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1272069332.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:48:24.098764Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:24.098793Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:24.105937Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1272069332.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:48:24.106026Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1272069332.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:48:24.106033Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1272069332.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:48:24.106051Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:24.106071Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:48:24.124359Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:48:24.124428Z info tf === SUCCEEDED: Cleanup Istio in 65.788µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3460864 2025-06-24T10:48:35.722447Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:48:35.722516Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:48:35.753367Z info tf === Test Framework Settings === 2025-06-24T10:48:35.753480Z info tf TestID: telemetry_tracing_otelcollector RunID: bcca38ed-c29c-4045-bee6-24c46afebeff NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:48:35.753486Z info tf =============================== 2025-06-24T10:48:35.753517Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:48:35.753561Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:48:35.753573Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:48:35.753592Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:48:35.753600Z info tf === BEGIN: Building clusters === 2025-06-24T10:48:35.756720Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:48:35.756735Z info tf === DONE: Building clusters === 2025-06-24T10:48:35.756765Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:48:35.756806Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:48:35.756817Z info tf === Istio Component Config === 2025-06-24T10:48:35.756928Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:48:35.756939Z info tf ================================ 2025-06-24T10:48:35.761561Z info tf skipping deployment as specified in the config 2025-06-24T10:48:35.761585Z info tf === SUCCEEDED: Deploy Istio in 4.773503ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:48:35.761595Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:48:35.788325Z info tf === SUCCEEDED: Create namespace echo in 26.711496ms === 2025-06-24T10:48:35.841214Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:48:36.037449Z info klog Waiting for caches to sync for echo 2025-06-24T10:48:36.083374Z info klog Waiting for caches to sync for echo 2025-06-24T10:48:36.138177Z info klog Caches are synced for echo 2025-06-24T10:48:36.184368Z info klog Caches are synced for echo 2025-06-24T10:48:42.489027Z info tf === SUCCEEDED: Deploy echo instances in 6.647798952s === 2025-06-24T10:48:42.523121Z info tf Checking pods ready... 2025-06-24T10:48:42.523182Z info tf Checking pods ready... 2025-06-24T10:48:42.527671Z info tf [ 0] zipkin-77cbd75c85-sgk7s Pending (Pending) 2025-06-24T10:48:42.727773Z info tf Checking pods ready... 2025-06-24T10:48:42.727839Z info tf Checking pods ready... 2025-06-24T10:48:42.732663Z info tf [ 0] zipkin-77cbd75c85-sgk7s Pending (Pending) 2025-06-24T10:48:43.133403Z info tf Checking pods ready... 2025-06-24T10:48:43.133487Z info tf Checking pods ready... 2025-06-24T10:48:43.137041Z info tf [ 0] zipkin-77cbd75c85-sgk7s Pending (Pending) 2025-06-24T10:48:43.937158Z info tf Checking pods ready... 2025-06-24T10:48:43.937208Z info tf Checking pods ready... 2025-06-24T10:48:43.940892Z info tf [ 0] zipkin-77cbd75c85-sgk7s Running (container not ready: 'zipkin') 2025-06-24T10:48:45.541814Z info tf Checking pods ready... 2025-06-24T10:48:45.541848Z info tf Checking pods ready... 2025-06-24T10:48:45.545563Z info tf [ 0] zipkin-77cbd75c85-sgk7s Running (container not ready: 'zipkin') 2025-06-24T10:48:48.746156Z info tf Checking pods ready... 2025-06-24T10:48:48.746302Z info tf Checking pods ready... 2025-06-24T10:48:48.750304Z info tf [ 0] zipkin-77cbd75c85-sgk7s Running (Ready) 2025-06-24T10:48:48.844049Z info tf Checking pods ready... 2025-06-24T10:48:48.844077Z info tf Checking pods ready... 2025-06-24T10:48:48.847348Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T10:48:49.047933Z info tf Checking pods ready... 2025-06-24T10:48:49.048035Z info tf Checking pods ready... 2025-06-24T10:48:49.052426Z info tf [ 0] opentelemetry-collector-854b84c844-rnnk4 Pending (Pending) 2025-06-24T10:48:49.452560Z info tf Checking pods ready... 2025-06-24T10:48:49.452619Z info tf Checking pods ready... 2025-06-24T10:48:49.457404Z info tf [ 0] opentelemetry-collector-854b84c844-rnnk4 Pending (Pending) 2025-06-24T10:48:50.258352Z info tf Checking pods ready... 2025-06-24T10:48:50.258429Z info tf Checking pods ready... 2025-06-24T10:48:50.262459Z info tf [ 0] opentelemetry-collector-854b84c844-rnnk4 Running (Ready) 2025-06-24T10:48:50.299523Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (14.542738256s) === 2025-06-24T10:48:50.299569Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 2025-06-24T10:48:50.308621Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:51:20.309240Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000616399s)' === 2025-06-24T10:51:20.309297Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:51:20.309302Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:51:20.461163Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66978d1fbd-2022-4f00-91b3-d388aaef4199 2025-06-24T10:51:20.461695Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1520s 2025-06-24T10:51:20.462192Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:20.546532Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/665808f174-edad-4019-b2d2-721b13154b99 2025-06-24T10:51:20.546646Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2369s 2025-06-24T10:51:20.547183Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:20.644097Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66afc79c77-0131-4838-9896-57ef75724881 2025-06-24T10:51:20.644398Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3347s 2025-06-24T10:51:20.644991Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (150.58s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 2) (150.58s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata 2025-06-24T10:48:50.300132Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T10:51:20.890348Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.590213781s)' === 2025-06-24T10:51:20.890468Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:51:20.890481Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:51:21.193050Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/660aa05e11-3b21-470d-bd83-117f42472ed0 2025-06-24T10:51:21.193192Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3025s 2025-06-24T10:51:21.193754Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:21.290395Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66493b24fa-e9bd-4b9c-af62-0baaa06e3bfd 2025-06-24T10:51:21.290755Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4000s 2025-06-24T10:51:21.291133Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:21.383480Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66cf8b282a-65a6-4c2e-8616-c095f8850deb 2025-06-24T10:51:21.383641Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4929s 2025-06-24T10:51:21.384302Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (151.31s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 2) (151.31s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T10:48:50.299937Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:49:48.204907Z error port forward failed: lost connection to pod 2025-06-24T10:49:48.695066Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward terminated 2025-06-24T10:49:48.726077Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward reconnect success 2025-06-24T10:49:48.779082Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:50:54.729719Z error port forward failed: lost connection to pod 2025-06-24T10:50:55.194732Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward terminated 2025-06-24T10:50:55.237791Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:51:21.612899Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.312958885s)' === 2025-06-24T10:51:21.612982Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:51:21.613019Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:51:21.769858Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67ae3db5be-f33e-488c-af41-1e7dfe73d8cd 2025-06-24T10:51:21.770240Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1570s 2025-06-24T10:51:21.770550Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:21.898879Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/673878c157-0b0c-41e6-9397-14fc13415622 2025-06-24T10:51:21.899069Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2858s 2025-06-24T10:51:21.899849Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:22.009235Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67407c836d-b4b4-435b-b20f-4d0b304dc923 2025-06-24T10:51:22.009392Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3961s 2025-06-24T10:51:22.009981Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (152.08s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 2) (152.08s) 2025-06-24T10:51:22.379530Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T10:51:22.379612Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m46.622855811s === 2025-06-24T10:51:22.379631Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T10:51:22.379905Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:51:22.379910Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:51:22.502403Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/679cb489ea-8d8b-473b-b9f3-4264829a1e1c 2025-06-24T10:51:22.502662Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1223s 2025-06-24T10:51:22.503249Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:22.588738Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67d208f76f-81ab-411a-ab2f-f4d156f8665d 2025-06-24T10:51:22.588889Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2085s 2025-06-24T10:51:22.589523Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:22.679457Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67dd4bdfbc-684d-4a18-9904-af723cc1a10e 2025-06-24T10:51:22.679725Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2993s 2025-06-24T10:51:22.680191Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:23.084367Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:51:23.084596Z info tf === SUCCEEDED: Cleanup Istio in 201.775µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3462263 2025-06-24T10:51:33.652838Z warn unable to resolve TARGET_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:51:33.652894Z warn unable to resolve LOCAL_OUT. Dir /root/swapnali/check_3.0_tp2/istio/out/linux_ppc64le does not exist 2025-06-24T10:51:33.686690Z info tf === Test Framework Settings === 2025-06-24T10:51:33.686784Z info tf TestID: telemetry_tracing_otelcollector RunID: 286e5423-66c5-483b-83d8-a6e1894406bb NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:51:33.686793Z info tf =============================== 2025-06-24T10:51:33.686819Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T10:51:33.686850Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:51:33.686858Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:51:33.686989Z info tf Using KubeConfigs: [/root/upi/cM/auth/kubeconfig]. 2025-06-24T10:51:33.686997Z info tf === BEGIN: Building clusters === 2025-06-24T10:51:33.689134Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:51:33.689143Z info tf === DONE: Building clusters === 2025-06-24T10:51:33.689163Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T10:51:33.689191Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:51:33.689197Z info tf === Istio Component Config === 2025-06-24T10:51:33.689227Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:51:33.689231Z info tf ================================ 2025-06-24T10:51:33.694978Z info tf skipping deployment as specified in the config 2025-06-24T10:51:33.695039Z info tf === SUCCEEDED: Deploy Istio in 5.843074ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T10:51:33.695050Z info tf === BEGIN: Create namespace echo === 2025-06-24T10:51:33.709561Z info tf === SUCCEEDED: Create namespace echo in 14.506846ms === 2025-06-24T10:51:33.734630Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:51:33.739755Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:51:33.739790Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2891374046.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:51:33.739815Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:51:33.739837Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:51:33.794746Z error klog an error occurred forwarding 42491 -> 15000: error forwarding port 15000 to pod 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e, uid : port forward into network namespace "/var/run/netns/cfdb001a-2843-4328-afff-60a7451989a5": failed to connect to localhost:15000 inside namespace 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:51:33.798173Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:42491/config_dump?include_eds=true": EOF 2025-06-24T10:51:33.798238Z error port forward failed: lost connection to pod 2025-06-24T10:51:33.798831Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:42491/clusters": dial tcp 127.0.0.1:42491: connect: connection refused 2025-06-24T10:51:33.803683Z error klog an error occurred forwarding 35949 -> 15000: error forwarding port 15000 to pod 3103d4a130ed64ead802a1c7a410ce15af797114baf17c809e41a8d84805f429, uid : port forward into network namespace "/var/run/netns/887e3684-ffef-4589-85ce-11732fb86cb3": failed to connect to localhost:15000 inside namespace 3103d4a130ed64ead802a1c7a410ce15af797114baf17c809e41a8d84805f429: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:51:33.805212Z error port forward failed: lost connection to pod 2025-06-24T10:51:33.805208Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-sp97w for: Get "http://localhost:35949/config_dump?include_eds=true": EOF 2025-06-24T10:51:33.805211Z error klog an error occurred forwarding 39743 -> 15020: error forwarding port 15020 to pod 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e, uid : port forward into network namespace "/var/run/netns/cfdb001a-2843-4328-afff-60a7451989a5": failed to connect to localhost:15020 inside namespace 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:51:33.805710Z error klog an error occurred forwarding 41253 -> 15020: error forwarding port 15020 to pod 3103d4a130ed64ead802a1c7a410ce15af797114baf17c809e41a8d84805f429, uid : port forward into network namespace "/var/run/netns/887e3684-ffef-4589-85ce-11732fb86cb3": failed to connect to localhost:15020 inside namespace 3103d4a130ed64ead802a1c7a410ce15af797114baf17c809e41a8d84805f429: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:51:33.805778Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-sp97w for: Get "http://localhost:35949/clusters": dial tcp 127.0.0.1:35949: connect: connection refused 2025-06-24T10:51:33.807378Z error port forward failed: lost connection to pod 2025-06-24T10:51:33.807411Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-sp97w for: Get "http://localhost:41253/debug/ndsz": EOF 2025-06-24T10:51:33.807458Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:39743/debug/ndsz": EOF 2025-06-24T10:51:33.807529Z error port forward failed: lost connection to pod 2025-06-24T10:51:33.807936Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:39743/stats/prometheus": dial tcp 127.0.0.1:39743: connect: connection refused 2025-06-24T10:51:33.808072Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-77c98755c8-sp97w for: Get "http://localhost:41253/stats/prometheus": dial tcp 127.0.0.1:41253: connect: connection refused 2025-06-24T10:51:33.836540Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:28Z/69530d70a1-e13a-4a43-b343-d9aeb54c1d47 2025-06-24T10:51:33.836666Z info tf istioctl ([x internal-debug --all configz]): completed after 0.0967s 2025-06-24T10:51:33.837127Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:33.924824Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:28Z/699cbeed9e-ba67-4a08-875d-39f16d9ce663 2025-06-24T10:51:33.924959Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1849s 2025-06-24T10:51:33.925434Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:34.011609Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:28Z/69c5fbf6dd-4606-4a98-be69-c1b9ab560946 2025-06-24T10:51:34.011760Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2717s 2025-06-24T10:51:34.012208Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:34.107445Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2891374046.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:51:34.107467Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2891374046.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T10:51:34.107474Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2891374046.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T10:51:34.107494Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:51:34.107494Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T10:51:34.159272Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.165953Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.172142Z error klog an error occurred forwarding 33603 -> 15020: error forwarding port 15020 to pod 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e, uid : port forward into network namespace "/var/run/netns/cfdb001a-2843-4328-afff-60a7451989a5": failed to connect to localhost:15020 inside namespace 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:51:34.173667Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:33603/debug/ndsz": EOF 2025-06-24T10:51:34.173740Z error port forward failed: lost connection to pod 2025-06-24T10:51:34.174251Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:33603/stats/prometheus": dial tcp 127.0.0.1:33603: connect: connection refused 2025-06-24T10:51:34.176240Z error klog an error occurred forwarding 35571 -> 15000: error forwarding port 15000 to pod 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e, uid : port forward into network namespace "/var/run/netns/cfdb001a-2843-4328-afff-60a7451989a5": failed to connect to localhost:15000 inside namespace 0fa6a639e36c7db49593622717d5ce9aa0d021133a75a50118dcb69c1aa7541e: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:51:34.178015Z error port forward failed: lost connection to pod 2025-06-24T10:51:34.178025Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:35571/config_dump?include_eds=true": EOF 2025-06-24T10:51:34.178658Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-7d889f5bc4-jmvf8 for: Get "http://localhost:35571/clusters": dial tcp 127.0.0.1:35571: connect: connection refused 2025-06-24T10:51:34.211288Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.217679Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.227853Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:28Z/69cd836575-b740-4e79-9302-0f2a91a085c0 2025-06-24T10:51:34.228001Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1203s 2025-06-24T10:51:34.228507Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:34.254341Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.260169Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.321078Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.326178Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.369227Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.373723Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:28Z/6966c7f34d-999b-4932-bf8a-39de1b398942 2025-06-24T10:51:34.373850Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2661s 2025-06-24T10:51:34.374511Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:34.376369Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo") 2025-06-24T10:51:34.379547Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo")) 2025-06-24T10:51:34.386778Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-77c98755c8-sp97w_echo")) 2025-06-24T10:51:34.478807Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:28Z/69d7f47db0-03dc-48bf-b78e-1ea8d3a0adc8 2025-06-24T10:51:34.478964Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3712s 2025-06-24T10:51:34.479548Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:34.602363Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T10:51:34.602444Z info tf === SUCCEEDED: Cleanup Istio in 76.304µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector === Failed === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (150.62s) 2025-06-24T10:24:24.281983Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:26:54.283120Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.001130896s)' === 2025-06-24T10:26:54.283187Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T10:26:54.283194Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T10:26:54.398185Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/9465ab84d-21c1-4381-967d-ba7581229b1a 2025-06-24T10:26:54.398629Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1149s 2025-06-24T10:26:54.398907Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:54.483642Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/91ba2bb76-12c7-46ca-941e-88076948453f 2025-06-24T10:26:54.483756Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2000s 2025-06-24T10:26:54.484311Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:54.572648Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/92a8a4891-f463-44d5-b4a9-6ee6e9dc1570 2025-06-24T10:26:54.572805Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2890s 2025-06-24T10:26:54.573285Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter (151.11s) 2025-06-24T10:24:24.268997Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T10:26:54.898091Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m30.629090776s)' === 2025-06-24T10:26:54.898164Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T10:26:54.898167Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T10:26:55.003003Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/9e4e438b6-96dc-43bd-9530-f416468b5286 2025-06-24T10:26:55.003330Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1050s 2025-06-24T10:26:55.003824Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:55.100949Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/93ac9ea10-b906-49de-9fd2-bfabfb62ee70 2025-06-24T10:26:55.101108Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2027s 2025-06-24T10:26:55.101557Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:55.187909Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:24:24Z/91367b36a-05bc-4838-8e5f-c0b0a6da7f07 2025-06-24T10:26:55.188175Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2898s 2025-06-24T10:26:55.188766Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (150.60s) 2025-06-24T10:26:55.396624Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:29:25.396941Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000314196s)' === 2025-06-24T10:29:25.396985Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:29:25.397182Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:29:25.620678Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10b0231a58-bdd0-4f8a-a2fb-0af6042a4f9f 2025-06-24T10:29:25.621064Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2231s 2025-06-24T10:29:25.621516Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:25.706044Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10041866f4-3a00-42c8-ab36-e38b23a8e176 2025-06-24T10:29:25.706187Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3082s 2025-06-24T10:29:25.706856Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:25.791038Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/1088901462-5d2f-4384-a2a4-ca839b46ab1a 2025-06-24T10:29:25.791219Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3933s 2025-06-24T10:29:25.791777Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter (151.68s) 2025-06-24T10:26:55.380398Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T10:29:25.999641Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.619239947s)' === 2025-06-24T10:29:25.999741Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:29:25.999750Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:29:26.228611Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10d5c5ecf4-b095-444e-8ac7-c372733689e5 2025-06-24T10:29:26.228932Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2290s 2025-06-24T10:29:26.229527Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:26.321146Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10cbe047d8-c021-40a9-ac90-6e68caf748a5 2025-06-24T10:29:26.321435Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3215s 2025-06-24T10:29:26.321878Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:29:26.404807Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:55Z/10d9fb423a-aea5-4760-bb1d-5a87be777708 2025-06-24T10:29:26.404967Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4050s 2025-06-24T10:29:26.405479Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (151.29s) 2025-06-24T10:29:27.077662Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:31:57.078651Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.000984655s)' === 2025-06-24T10:31:57.078795Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:31:57.078807Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:31:57.247984Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11033878dd-bb37-4d18-8109-7418280443c1 2025-06-24T10:31:57.248278Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1657s 2025-06-24T10:31:57.249007Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:57.536615Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/111b5777a1-9f22-45ba-a717-93dfbc5b4e79 2025-06-24T10:31:57.536758Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4541s 2025-06-24T10:31:57.537531Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:57.648310Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11fb606a1c-ed1b-4841-9b44-344fcbad8c15 2025-06-24T10:31:57.648456Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5658s 2025-06-24T10:31:57.649135Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors (152.45s) 2025-06-24T10:29:27.065836Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T10:31:58.364632Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m31.298795927s)' === 2025-06-24T10:31:58.364672Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:31:58.364678Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:31:58.486489Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11974b34de-640c-461b-aae0-a94bf7aa98ca 2025-06-24T10:31:58.486828Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1211s 2025-06-24T10:31:58.487362Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:58.579387Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/11cd2aac99-8dbd-4764-ba69-6fa2546a0d37 2025-06-24T10:31:58.579547Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2138s 2025-06-24T10:31:58.580147Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:31:58.677749Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:29:27Z/111c026da9-cd8a-4882-9a42-14d4c6659ac2 2025-06-24T10:31:58.677969Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3121s 2025-06-24T10:31:58.678569Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (151.31s) 2025-06-24T10:31:59.532805Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:34:29.533272Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000465483s)' === 2025-06-24T10:34:29.533352Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:34:29.533345Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:34:29.813266Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/127be79e69-453c-4852-bf46-10964075c52f 2025-06-24T10:34:29.813688Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2795s 2025-06-24T10:34:29.814176Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:29.913795Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/121f58ef3e-771c-4cbe-ac10-d0d52afb66b4 2025-06-24T10:34:29.914008Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3799s 2025-06-24T10:34:29.914636Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:30.031278Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/126917b8ad-1e62-4bad-96a2-cdaa5ac8c8ca 2025-06-24T10:34:30.031475Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4973s 2025-06-24T10:34:30.032082Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (152.77s) 2025-06-24T10:31:59.519056Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T10:34:30.846714Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m31.327651868s)' === 2025-06-24T10:34:30.846866Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:34:30.846863Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:34:30.972360Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/12e3d0fd95-20db-4511-8384-45a15c18c5cb 2025-06-24T10:34:30.972565Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1254s 2025-06-24T10:34:30.973315Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:31.074305Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/12e0ef809f-cd41-42d8-939a-33a7ad1ef1e8 2025-06-24T10:34:31.074421Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2273s 2025-06-24T10:34:31.074934Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:31.171566Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:31:59Z/12de6cc5d9-21cc-4993-8979-cdc54c243162 2025-06-24T10:34:31.172019Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3248s 2025-06-24T10:34:31.172323Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (609.58s) 2025-06-24T10:24:24.268913Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:24:35.351298Z error port forward failed: lost connection to pod 2025-06-24T10:24:35.588028Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:24:35.620946Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:25:11.512353Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:25:41.625366Z error port forward failed: lost connection to pod 2025-06-24T10:25:42.088199Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:25:42.155637Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:26:48.160566Z error port forward failed: lost connection to pod 2025-06-24T10:26:48.587320Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:26:48.685981Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:27:54.690429Z error port forward failed: lost connection to pod 2025-06-24T10:27:55.088305Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:27:55.125242Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:29:01.130102Z error port forward failed: lost connection to pod 2025-06-24T10:29:01.587651Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:29:01.625602Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:30:07.634263Z error port forward failed: lost connection to pod 2025-06-24T10:30:08.088253Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:30:08.134770Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:31:14.138454Z error port forward failed: lost connection to pod 2025-06-24T10:31:14.588175Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward terminated 2025-06-24T10:31:14.620522Z warn pod: echo/server-v1-7d889f5bc4-2g84m port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:32:20.625308Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:34:32.293470Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (10m8.024554033s)' === 2025-06-24T10:34:32.293548Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:34:32.293546Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:34:32.422810Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/13c405b575-90e8-4d82-bd1c-35727d90b98a 2025-06-24T10:34:32.423211Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1288s 2025-06-24T10:34:32.423662Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:32.525598Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/136f39b33d-ea4f-4c43-aa3f-6fff094214ca 2025-06-24T10:34:32.525774Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2313s 2025-06-24T10:34:32.526508Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:34:32.625256Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:34:32Z/13d0793073-c611-4f1a-ae25-f459f036057e 2025-06-24T10:34:32.625439Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3309s 2025-06-24T10:34:32.625965Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 1) (150.59s) 2025-06-24T10:35:34.162344Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:38:04.162705Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000355161s)' === 2025-06-24T10:38:04.162876Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:38:04.162863Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:38:04.300867Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/2345e056f7-a567-463b-91a9-8f5b195baad4 2025-06-24T10:38:04.301246Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1375s 2025-06-24T10:38:04.301668Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:04.400020Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/239b951696-c17c-4228-8289-184e72a65982 2025-06-24T10:38:04.400093Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2364s 2025-06-24T10:38:04.400733Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:04.509781Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/234c485a29-df6a-436b-a9b6-f73477c8e68e 2025-06-24T10:38:04.509909Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3462s 2025-06-24T10:38:04.510649Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 1) (151.15s) 2025-06-24T10:35:34.150392Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T10:38:04.749249Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.598852783s)' === 2025-06-24T10:38:04.749347Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:38:04.749350Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:38:04.913542Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/23e80e1b30-b167-4a34-b46c-dc145358b13a 2025-06-24T10:38:04.913663Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1641s 2025-06-24T10:38:04.914341Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.037332Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/239b552278-90db-427c-99e7-2564c573473d 2025-06-24T10:38:05.037440Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2878s 2025-06-24T10:38:05.038123Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.141178Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:35:34Z/231268314c-03a9-4a0d-928c-0627606c693f 2025-06-24T10:38:05.141539Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3919s 2025-06-24T10:38:05.141873Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 1) (151.98s) 2025-06-24T10:35:34.150131Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:36:23.674151Z error port forward failed: lost connection to pod 2025-06-24T10:36:24.043612Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward terminated 2025-06-24T10:36:24.075118Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:36:31.001401Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:37:30.079287Z error port forward failed: lost connection to pod 2025-06-24T10:37:30.544165Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward terminated 2025-06-24T10:37:30.575197Z warn pod: echo/server-v1-7d889f5bc4-v88s2 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:38:05.301371Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.151232748s)' === 2025-06-24T10:38:05.301479Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:38:05.301479Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:38:05.434661Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/240bbab46b-d7c1-41cc-9443-95991355aaec 2025-06-24T10:38:05.434966Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1331s 2025-06-24T10:38:05.435374Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.538242Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/247e96b69d-ed41-45a9-b44e-3f33d91af6d5 2025-06-24T10:38:05.538341Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2365s 2025-06-24T10:38:05.538901Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:38:05.628215Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:05Z/24b107e107-b592-4d58-aefc-255ff2263a6a 2025-06-24T10:38:05.628400Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3265s 2025-06-24T10:38:05.629132Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (re-run 1) (150.46s) 2025-06-24T10:38:47.264152Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:41:17.265065Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.000907014s)' === 2025-06-24T10:41:17.265156Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:41:17.265145Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T10:41:17.362511Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/3362c324be-a103-4e97-8564-8c32db4910ce 2025-06-24T10:41:17.363310Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:17.363369Z info tf istioctl ([x internal-debug --all configz]): completed after 0.0978s 2025-06-24T10:41:17.459225Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/3303b247eb-e5d4-4ab1-af69-1c5944fdba66 2025-06-24T10:41:17.459389Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1938s 2025-06-24T10:41:17.460007Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:17.544615Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/33aec11ba3-15e7-4097-9420-849e81bf42e8 2025-06-24T10:41:17.544756Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2792s 2025-06-24T10:41:17.545396Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors (re-run 1) (151.05s) 2025-06-24T10:38:47.256240Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T10:41:17.725570Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m30.46932487s)' === 2025-06-24T10:41:17.725639Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:41:17.725656Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T10:41:17.846621Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/33f32bb676-1f46-4da3-9ea2-cc03c909df2f 2025-06-24T10:41:17.846993Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1212s 2025-06-24T10:41:17.847506Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:17.934788Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/33911bd887-74f3-4e15-bcce-e9a1e28a7d78 2025-06-24T10:41:17.934897Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2091s 2025-06-24T10:41:17.935383Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:18.021524Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:38:47Z/3339a62912-e6c9-431b-8313-6fe301c48ed0 2025-06-24T10:41:18.021702Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2958s 2025-06-24T10:41:18.022211Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 1) (151.76s) 2025-06-24T10:38:47.256088Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:39:42.684649Z error port forward failed: lost connection to pod 2025-06-24T10:39:42.959446Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward terminated 2025-06-24T10:39:42.996520Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward reconnect success 2025-06-24T10:39:44.113700Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:40:49.000670Z error port forward failed: lost connection to pod 2025-06-24T10:40:49.459565Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward terminated 2025-06-24T10:40:49.490458Z warn pod: echo/server-v1-7d889f5bc4-7cnsh port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:41:18.305841Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.049748776s)' === 2025-06-24T10:41:18.305932Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:41:18.305950Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:41:18.426381Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/34c62ba335-8a74-4965-ab7d-386a5dc18795 2025-06-24T10:41:18.426790Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1206s 2025-06-24T10:41:18.427289Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:18.510144Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/3406476c30-02b9-4830-a5e9-b07363946c57 2025-06-24T10:41:18.510403Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2042s 2025-06-24T10:41:18.510954Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:41:18.603368Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:18Z/34e089c52a-b480-460a-9f10-2fb43701041c 2025-06-24T10:41:18.603575Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2974s 2025-06-24T10:41:18.604124Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 1) (150.45s) 2025-06-24T10:41:58.196552Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:44:28.196941Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000379685s)' === 2025-06-24T10:44:28.197058Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:44:28.197060Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:44:28.312919Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/44b7454c35-080a-4aab-88bc-0589eb888154 2025-06-24T10:44:28.313055Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1156s 2025-06-24T10:44:28.313635Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.405816Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/44becb45cf-690a-41e4-9cbb-88c7763ed25f 2025-06-24T10:44:28.406104Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2086s 2025-06-24T10:44:28.406581Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.498958Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/441fbd5eb0-4a30-4538-97ef-3fee68bb40d3 2025-06-24T10:44:28.499098Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3016s 2025-06-24T10:44:28.499735Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 1) (150.97s) 2025-06-24T10:41:58.188771Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T10:44:28.643229Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.454454386s)' === 2025-06-24T10:44:28.643311Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:44:28.643322Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:44:28.760454Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/44018f0064-0043-4f0a-8f4a-02bb18d244a3 2025-06-24T10:44:28.760737Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1173s 2025-06-24T10:44:28.761221Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.851166Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/447beeb700-0d89-46e1-b4de-023c07fe7696 2025-06-24T10:44:28.851368Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2079s 2025-06-24T10:44:28.851984Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:28.936515Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:41:58Z/447ff678fa-9308-4a53-90ee-5e00bfa5e283 2025-06-24T10:44:28.936666Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2932s 2025-06-24T10:44:28.937142Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 1) (151.68s) 2025-06-24T10:41:58.188542Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:42:54.664830Z error port forward failed: lost connection to pod 2025-06-24T10:42:54.840355Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward terminated 2025-06-24T10:42:54.876821Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward reconnect success 2025-06-24T10:42:54.999665Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:44:00.881841Z error port forward failed: lost connection to pod 2025-06-24T10:44:01.339993Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward terminated 2025-06-24T10:44:01.381850Z warn pod: echo/server-v1-7d889f5bc4-bmfm9 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:44:29.157047Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m30.968502163s)' === 2025-06-24T10:44:29.157111Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:44:29.157110Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:44:29.280802Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/4534137018-3643-4ebb-ba87-bb23a484224b 2025-06-24T10:44:29.281103Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1237s 2025-06-24T10:44:29.281647Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:29.361759Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/4575c6fedb-173e-4f82-9657-cc2f97d64eb7 2025-06-24T10:44:29.361927Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2046s 2025-06-24T10:44:29.362477Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:44:29.450245Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:44:29Z/4593de1097-44f8-420e-af7e-d29729694b97 2025-06-24T10:44:29.450512Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2931s 2025-06-24T10:44:29.450966Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 2) (150.45s) 2025-06-24T10:45:28.039767Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:47:58.040488Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000717748s)' === 2025-06-24T10:47:58.040588Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:47:58.040588Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T10:47:58.159384Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/555ada0cec-48ec-45b3-88cc-3a31a7e63ae6 2025-06-24T10:47:58.159578Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1184s 2025-06-24T10:47:58.160197Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.246024Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/5579f7f84d-dcbe-423b-830a-9e06f59e5fae 2025-06-24T10:47:58.246145Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2049s 2025-06-24T10:47:58.246747Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.385060Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/55c733a5b3-a3f0-400d-a4a2-8483997cdf0e 2025-06-24T10:47:58.385466Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3443s 2025-06-24T10:47:58.385974Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 2) (150.94s) 2025-06-24T10:45:28.031126Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T10:47:58.491625Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.460496547s)' === 2025-06-24T10:47:58.491687Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:47:58.491689Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T10:47:58.595966Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/553c7c52a8-924e-408d-a876-9c3ad1e303b9 2025-06-24T10:47:58.596232Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1044s 2025-06-24T10:47:58.596649Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.684805Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/55dc7ee313-3ec1-4d69-9f3a-c00f4982d35f 2025-06-24T10:47:58.684973Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1931s 2025-06-24T10:47:58.685575Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:58.770739Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:45:28Z/55c9e8d6f5-e7e6-4252-8f89-3b3996a5ff94 2025-06-24T10:47:58.770878Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2790s 2025-06-24T10:47:58.771355Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 2) (151.59s) 2025-06-24T10:45:28.030968Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:46:16.801049Z error port forward failed: lost connection to pod 2025-06-24T10:46:16.826567Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward terminated 2025-06-24T10:46:16.860829Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:46:24.897306Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:47:22.864512Z error port forward failed: lost connection to pod 2025-06-24T10:47:23.326351Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward terminated 2025-06-24T10:47:23.359151Z warn pod: echo/server-v1-7d889f5bc4-6l4jj port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:47:58.975232Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m30.944257413s)' === 2025-06-24T10:47:58.975337Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:47:58.975330Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:47:59.084698Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/561c702580-a6fc-40b6-96e1-1652545ca728 2025-06-24T10:47:59.085134Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1096s 2025-06-24T10:47:59.085543Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:59.191186Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/564fe66857-201a-4958-b828-26a12c6d7959 2025-06-24T10:47:59.191343Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2157s 2025-06-24T10:47:59.191961Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:47:59.289805Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:47:59Z/56505d8648-48b7-472b-95ec-d5b6a7793c5f 2025-06-24T10:47:59.290201Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3146s 2025-06-24T10:47:59.290799Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 2) (150.58s) 2025-06-24T10:48:50.308621Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T10:51:20.309240Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000616399s)' === 2025-06-24T10:51:20.309297Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:51:20.309302Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T10:51:20.461163Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66978d1fbd-2022-4f00-91b3-d388aaef4199 2025-06-24T10:51:20.461695Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1520s 2025-06-24T10:51:20.462192Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:20.546532Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/665808f174-edad-4019-b2d2-721b13154b99 2025-06-24T10:51:20.546646Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2369s 2025-06-24T10:51:20.547183Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:20.644097Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66afc79c77-0131-4838-9896-57ef75724881 2025-06-24T10:51:20.644398Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3347s 2025-06-24T10:51:20.644991Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 2) (151.31s) 2025-06-24T10:48:50.300132Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T10:51:20.890348Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.590213781s)' === 2025-06-24T10:51:20.890468Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:51:20.890481Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T10:51:21.193050Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/660aa05e11-3b21-470d-bd83-117f42472ed0 2025-06-24T10:51:21.193192Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3025s 2025-06-24T10:51:21.193754Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:21.290395Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66493b24fa-e9bd-4b9c-af62-0baaa06e3bfd 2025-06-24T10:51:21.290755Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4000s 2025-06-24T10:51:21.291133Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:21.383480Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:48:50Z/66cf8b282a-65a6-4c2e-8616-c095f8850deb 2025-06-24T10:51:21.383641Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4929s 2025-06-24T10:51:21.384302Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 2) (152.08s) 2025-06-24T10:48:50.299937Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:49:48.204907Z error port forward failed: lost connection to pod 2025-06-24T10:49:48.695066Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward terminated 2025-06-24T10:49:48.726077Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward reconnect success 2025-06-24T10:49:48.779082Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:50:54.729719Z error port forward failed: lost connection to pod 2025-06-24T10:50:55.194732Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward terminated 2025-06-24T10:50:55.237791Z warn pod: echo/server-v1-7d889f5bc4-jmvf8 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cM/auth/kubeconfig 2025-06-24T10:51:21.612899Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.312958885s)' === 2025-06-24T10:51:21.612982Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:51:21.613019Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T10:51:21.769858Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67ae3db5be-f33e-488c-af41-1e7dfe73d8cd 2025-06-24T10:51:21.770240Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1570s 2025-06-24T10:51:21.770550Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:21.898879Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/673878c157-0b0c-41e6-9397-14fc13415622 2025-06-24T10:51:21.899069Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2858s 2025-06-24T10:51:21.899849Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:51:22.009235Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:51:21Z/67407c836d-b4b4-435b-b20f-4d0b304dc923 2025-06-24T10:51:22.009392Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3961s 2025-06-24T10:51:22.009981Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 3 runs, 24 tests, 24 failures in 1716.047s exec: go version [root@cpt-ppc-059 istio]#