exec: [go test -json -tags=integ -timeout 180m ./tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3538062 2025-06-24T15:35:16.274835Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:35:16.274907Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:35:16.304273Z info tf === Test Framework Settings === 2025-06-24T15:35:16.304355Z info tf TestID: telemetry_tracing_otelcollector RunID: 2fb58a08-8b99-46a3-a128-1f5850cf1cb5 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:35:16.304361Z info tf =============================== 2025-06-24T15:35:16.306017Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:35:16.306338Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:35:16.306350Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:35:16.306512Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:35:16.306522Z info tf === BEGIN: Building clusters === 2025-06-24T15:35:16.308841Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:35:16.308848Z info tf === DONE: Building clusters === 2025-06-24T15:35:16.308862Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:35:16.309223Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:35:16.309232Z info tf === Istio Component Config === 2025-06-24T15:35:16.309261Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:35:16.309266Z info tf ================================ 2025-06-24T15:35:16.315247Z info tf skipping deployment as specified in the config 2025-06-24T15:35:16.315326Z info tf === SUCCEEDED: Deploy Istio in 6.095779ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:35:16.315333Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:35:16.337326Z info tf === SUCCEEDED: Create namespace echo in 21.986597ms === 2025-06-24T15:35:16.423224Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:35:16.866597Z info klog Waiting for caches to sync for echo 2025-06-24T15:35:16.866711Z info klog Waiting for caches to sync for echo 2025-06-24T15:35:16.967276Z info klog Caches are synced for echo 2025-06-24T15:35:16.967330Z info klog Caches are synced for echo 2025-06-24T15:35:28.074891Z info tf === SUCCEEDED: Deploy echo instances in 11.651653411s === 2025-06-24T15:35:28.227181Z info tf Checking pods ready... 2025-06-24T15:35:28.227228Z info tf Checking pods ready... 2025-06-24T15:35:28.251528Z info tf [ 0] zipkin-78496844d5-6zrj5 Pending (Pending) 2025-06-24T15:35:28.452303Z info tf Checking pods ready... 2025-06-24T15:35:28.452339Z info tf Checking pods ready... 2025-06-24T15:35:28.459027Z info tf [ 0] zipkin-78496844d5-6zrj5 Pending (Pending) 2025-06-24T15:35:28.860021Z info tf Checking pods ready... 2025-06-24T15:35:28.860094Z info tf Checking pods ready... 2025-06-24T15:35:28.864103Z info tf [ 0] zipkin-78496844d5-6zrj5 Pending (Pending) 2025-06-24T15:35:29.664696Z info tf Checking pods ready... 2025-06-24T15:35:29.664806Z info tf Checking pods ready... 2025-06-24T15:35:29.674107Z info tf [ 0] zipkin-78496844d5-6zrj5 Pending (Pending) 2025-06-24T15:35:31.274596Z info tf Checking pods ready... 2025-06-24T15:35:31.274691Z info tf Checking pods ready... 2025-06-24T15:35:31.279590Z info tf [ 0] zipkin-78496844d5-6zrj5 Pending (Pending) 2025-06-24T15:35:34.480011Z info tf Checking pods ready... 2025-06-24T15:35:34.480059Z info tf Checking pods ready... 2025-06-24T15:35:34.484576Z info tf [ 0] zipkin-78496844d5-6zrj5 Pending (Pending) 2025-06-24T15:35:37.685641Z info tf Checking pods ready... 2025-06-24T15:35:37.685680Z info tf Checking pods ready... 2025-06-24T15:35:37.703269Z info tf [ 0] zipkin-78496844d5-6zrj5 Pending (Pending) 2025-06-24T15:35:40.903343Z info tf Checking pods ready... 2025-06-24T15:35:40.903411Z info tf Checking pods ready... 2025-06-24T15:35:40.920530Z info tf [ 0] zipkin-78496844d5-6zrj5 Running (container not ready: 'zipkin') 2025-06-24T15:35:44.121268Z info tf Checking pods ready... 2025-06-24T15:35:44.121315Z info tf Checking pods ready... 2025-06-24T15:35:44.125953Z info tf [ 0] zipkin-78496844d5-6zrj5 Running (Ready) 2025-06-24T15:35:44.271635Z info tf Checking pods ready... 2025-06-24T15:35:44.271661Z info tf Checking pods ready... 2025-06-24T15:35:44.284036Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T15:35:44.484851Z info tf Checking pods ready... 2025-06-24T15:35:44.484936Z info tf Checking pods ready... 2025-06-24T15:35:44.490196Z info tf [ 0] opentelemetry-collector-69596cbbd6-wvwjh Pending (Pending) 2025-06-24T15:35:44.891263Z info tf Checking pods ready... 2025-06-24T15:35:44.891317Z info tf Checking pods ready... 2025-06-24T15:35:44.896046Z info tf [ 0] opentelemetry-collector-69596cbbd6-wvwjh Pending (Pending) 2025-06-24T15:35:45.696635Z info tf Checking pods ready... 2025-06-24T15:35:45.696710Z info tf Checking pods ready... 2025-06-24T15:35:45.704029Z info tf [ 0] opentelemetry-collector-69596cbbd6-wvwjh Pending (Pending) 2025-06-24T15:35:47.304918Z info tf Checking pods ready... 2025-06-24T15:35:47.304949Z info tf Checking pods ready... 2025-06-24T15:35:47.310399Z info tf [ 0] opentelemetry-collector-69596cbbd6-wvwjh Pending (Pending) 2025-06-24T15:35:50.510802Z info tf Checking pods ready... 2025-06-24T15:35:50.510853Z info tf Checking pods ready... 2025-06-24T15:35:50.522171Z info tf [ 0] opentelemetry-collector-69596cbbd6-wvwjh Pending (Pending) 2025-06-24T15:35:53.723085Z info tf Checking pods ready... 2025-06-24T15:35:53.723125Z info tf Checking pods ready... 2025-06-24T15:35:53.731210Z info tf [ 0] opentelemetry-collector-69596cbbd6-wvwjh Running (Ready) 2025-06-24T15:35:53.771218Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (37.462345227s) === 2025-06-24T15:35:53.771253Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 2025-06-24T15:35:53.781163Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:38:23.781622Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.000455019s)' === 2025-06-24T15:38:23.781713Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:38:23.781709Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:38:23.947198Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/12d286231e-945b-41df-98b8-387a7090060c 2025-06-24T15:38:23.947717Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1653s 2025-06-24T15:38:23.948095Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.039991Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/1290404357-75a6-41d0-809c-5afe42f52286 2025-06-24T15:38:24.040139Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2577s 2025-06-24T15:38:24.040632Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.124277Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/12523ed408-d217-4650-a9f8-45952ebfe7da 2025-06-24T15:38:24.124385Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3419s 2025-06-24T15:38:24.125031Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (150.69s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (150.69s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter 2025-06-24T15:35:53.771730Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T15:38:24.475702Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m30.703969388s)' === 2025-06-24T15:38:24.475775Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:38:24.475788Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:38:24.611695Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/129b79c1ff-69b3-4159-8e5a-458bb8c1e900 2025-06-24T15:38:24.612074Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1361s 2025-06-24T15:38:24.612437Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.706678Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/121cc6233f-c377-445b-9f55-6547b261d253 2025-06-24T15:38:24.706812Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2309s 2025-06-24T15:38:24.707413Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.811488Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/1207fd7836-2a3d-4255-8046-0fcb414769a1 2025-06-24T15:38:24.811662Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3356s 2025-06-24T15:38:24.812254Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter (151.56s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter (151.56s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 2025-06-24T15:38:25.344967Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:40:55.345947Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000976684s)' === 2025-06-24T15:40:55.346052Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:40:55.346315Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:40:56.317496Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/13abc8ecd7-59cf-4b56-803b-611388f0b9d1 2025-06-24T15:40:56.317788Z info tf istioctl ([x internal-debug --all configz]): completed after 0.9711s 2025-06-24T15:40:56.318274Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (151.14s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (151.14s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter 2025-06-24T15:38:25.331127Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T15:40:56.479480Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m31.148347397s)' === 2025-06-24T15:40:56.479573Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:40:56.479572Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:40:56.488755Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/13ded7bbed-77ef-43fd-a52c-4279d23df06f 2025-06-24T15:40:56.489508Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1427s 2025-06-24T15:40:56.492385Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:40:56.710901Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/133c764846-fb32-410c-97a7-80be85e3857b 2025-06-24T15:40:56.711272Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.3644s 2025-06-24T15:40:56.711741Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:40:57.127642Z error port forward failed: lost connection to pod 2025-06-24T15:40:57.164622Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/13dc807926-dc04-4f67-bea9-fd334247277d 2025-06-24T15:40:57.164906Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6851s 2025-06-24T15:40:57.165437Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter (151.96s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter (151.96s) === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 2025-06-24T15:40:57.300962Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:43:27.301537Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.000575062s)' === 2025-06-24T15:43:27.301654Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:43:27.301649Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:43:27.538617Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/144b0318a1-a3fa-4c6d-b578-026eee176981 2025-06-24T15:43:27.538734Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2364s 2025-06-24T15:43:27.539555Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:27.608720Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/14341c8c98-525c-4ee5-9bc9-8089c21785b4 2025-06-24T15:43:27.608946Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3066s 2025-06-24T15:43:27.609566Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:27.695701Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/14f655a400-756d-4436-90f7-1eac0d7296df 2025-06-24T15:43:27.695849Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3934s 2025-06-24T15:43:27.696622Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (150.94s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (150.94s) === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors 2025-06-24T15:40:57.288128Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T15:43:28.238319Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m30.950181002s)' === 2025-06-24T15:43:28.238436Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:43:28.238430Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:43:28.446980Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/14609cb876-3107-4116-b271-cac586fbe721 2025-06-24T15:43:28.447330Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2086s 2025-06-24T15:43:28.447783Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:28.570459Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/1499dde033-77cb-471d-81a4-91ce3f3744ac 2025-06-24T15:43:28.570677Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3319s 2025-06-24T15:43:28.571487Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:28.687799Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/142f7902c1-46b1-4fb3-b628-1e130a667793 2025-06-24T15:43:28.687972Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4492s 2025-06-24T15:43:28.688487Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors (152.05s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors (152.05s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 2025-06-24T15:43:29.354322Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:45:59.355303Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000980727s)' === 2025-06-24T15:45:59.355415Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:45:59.355424Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:45:59.553603Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15e874dedd-eda1-42fc-82cc-e7d3ef24f92d 2025-06-24T15:45:59.553970Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1974s 2025-06-24T15:45:59.554847Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:45:59.714043Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15b52a17fe-f863-439d-acec-cf9385e9094a 2025-06-24T15:45:59.714158Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3576s 2025-06-24T15:45:59.714896Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:45:59.797060Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/159fb613d2-b2ee-4ead-893a-fcbea8659252 2025-06-24T15:45:59.797190Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4406s 2025-06-24T15:45:59.797785Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (151.26s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (151.26s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata 2025-06-24T15:43:29.337087Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T15:46:00.617192Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m31.280099531s)' === 2025-06-24T15:46:00.617305Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:46:00.617304Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:46:00.746891Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15734335d3-02c4-41db-987f-994818f61adf 2025-06-24T15:46:00.747179Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1297s 2025-06-24T15:46:00.747590Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:00.879582Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/1585f7ba94-1e5f-418c-9056-922044316395 2025-06-24T15:46:00.879779Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2623s 2025-06-24T15:46:00.880354Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:00.956424Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15e8dfb61a-e8fe-462c-9d62-73ef46ef13c9 2025-06-24T15:46:00.956576Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3390s 2025-06-24T15:46:00.957061Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (152.43s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (152.43s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T15:35:53.771571Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:36:31.430751Z error port forward failed: lost connection to pod 2025-06-24T15:36:31.585073Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:36:31.631149Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:36:44.153686Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:37:37.635973Z error port forward failed: lost connection to pod 2025-06-24T15:37:38.085734Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:37:38.126855Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:38:44.132160Z error port forward failed: lost connection to pod 2025-06-24T15:38:44.586033Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:38:44.617740Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:39:50.622812Z error port forward failed: lost connection to pod 2025-06-24T15:39:51.085652Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:39:51.122708Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:40:57.322422Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/133c07ff59-6635-4d97-8b22-66e4ae37a32b 2025-06-24T15:40:57.322510Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8427s 2025-06-24T15:40:57.323137Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:40:57.444697Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/1487b53829-7799-4219-9cd1-a804ae258e7b 2025-06-24T15:40:57.444783Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9649s 2025-06-24T15:40:57.445346Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:46:01.765846Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (10m7.994269038s)' === 2025-06-24T15:46:01.765991Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:46:01.765993Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:46:01.925699Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/161d5ecbec-2c12-4d40-83c0-f2fa81adc02b 2025-06-24T15:46:01.925945Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1591s 2025-06-24T15:46:01.926700Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:02.107230Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/160e745e9b-566b-4a2c-9abf-7b2345977ee2 2025-06-24T15:46:02.107425Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3405s 2025-06-24T15:46:02.108065Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:02.202311Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/16e09e6789-4aae-490e-8c20-e4cdf175f888 2025-06-24T15:46:02.202569Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4356s 2025-06-24T15:46:02.203199Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (609.37s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (609.37s) 2025-06-24T15:46:03.145935Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T15:46:03.146032Z info tf === Suite "telemetry_tracing_otelcollector" run time: 10m46.837165682s === 2025-06-24T15:46:03.146055Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T15:46:03.147578Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:46:03.147602Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:46:03.324056Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/16303918fc-0cae-4966-a1e0-95f72d18b0d4 2025-06-24T15:46:03.324405Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1764s 2025-06-24T15:46:03.324886Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:03.407285Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/1619728040-3ea2-4ee2-a392-fdf3f5ba9b4c 2025-06-24T15:46:03.407493Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2595s 2025-06-24T15:46:03.408256Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:03.539707Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/16e3a8bbf2-5747-44c3-9d59-ca2bb704f390 2025-06-24T15:46:03.540056Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3920s 2025-06-24T15:46:03.540741Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:04.974707Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:46:04.974823Z info tf === SUCCEEDED: Cleanup Istio in 104.433µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector DONE 9 tests, 9 failures in 664.587s exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3540486 2025-06-24T15:46:16.034016Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:46:16.034070Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:46:16.063956Z info tf === Test Framework Settings === 2025-06-24T15:46:16.064038Z info tf TestID: telemetry_tracing_otelcollector RunID: c7f104e1-749d-4a6a-9a3d-68a694aef64c NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:46:16.064044Z info tf =============================== 2025-06-24T15:46:16.064074Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:46:16.064110Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:46:16.064123Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:46:16.064142Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:46:16.064150Z info tf === BEGIN: Building clusters === 2025-06-24T15:46:16.066640Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:46:16.066647Z info tf === DONE: Building clusters === 2025-06-24T15:46:16.066666Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:46:16.066687Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:46:16.066693Z info tf === Istio Component Config === 2025-06-24T15:46:16.066719Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:46:16.066723Z info tf ================================ 2025-06-24T15:46:16.069682Z info tf skipping deployment as specified in the config 2025-06-24T15:46:16.069696Z info tf === SUCCEEDED: Deploy Istio in 3.00546ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:46:16.069702Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:46:16.111586Z info tf === SUCCEEDED: Create namespace echo in 41.853675ms === 2025-06-24T15:46:16.151342Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:46:16.159512Z error tf === FAILED: Deploy echo instances === 2025-06-24T15:46:16.159561Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3482360328.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:46:16.159596Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:46:16.159595Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:46:16.221311Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-ghkg5_echo") 2025-06-24T15:46:16.224147Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-5c7cd449cf-8ht76_echo") 2025-06-24T15:46:16.240729Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-ghkg5_echo") 2025-06-24T15:46:16.279159Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-5c7cd449cf-8ht76_echo") 2025-06-24T15:46:16.281196Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-5c7cd449cf-8ht76_echo") 2025-06-24T15:46:16.356449Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:46:16.358745Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:10Z/183ce363ed-c444-4870-ad15-df7d8b127486 2025-06-24T15:46:16.358924Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1990s 2025-06-24T15:46:16.359516Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:16.362078Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:46:16.478081Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:10Z/183d039f7e-055b-436a-9d50-307f8b349a3f 2025-06-24T15:46:16.478323Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3184s 2025-06-24T15:46:16.478868Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:16.585017Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:10Z/185fcb8e1e-5c79-48cd-988f-506c98c82187 2025-06-24T15:46:16.585984Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4258s 2025-06-24T15:46:16.587033Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:17.196510Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-ghkg5/app for: pods "server-v1-6f45fcb457-ghkg5" not found 2025-06-24T15:46:17.202590Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-ghkg5/istio-proxy for: pods "server-v1-6f45fcb457-ghkg5" not found 2025-06-24T15:46:17.208075Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-ghkg5/istio-validation for: pods "server-v1-6f45fcb457-ghkg5" not found 2025-06-24T15:46:17.220530Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3482360328.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:46:17.220578Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3482360328.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T15:46:17.220587Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3482360328.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:46:17.220633Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:46:17.220624Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:46:17.231870Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:46:17.231986Z info tf === SUCCEEDED: Cleanup Istio in 103.642µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3541875 2025-06-24T15:46:27.727501Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:46:27.727556Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:46:27.760812Z info tf === Test Framework Settings === 2025-06-24T15:46:27.760902Z info tf TestID: telemetry_tracing_otelcollector RunID: be2f4b72-c2ad-4b8b-a0fd-12e08fbd276d NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:46:27.760909Z info tf =============================== 2025-06-24T15:46:27.760937Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:46:27.760979Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:46:27.760990Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:46:27.761006Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:46:27.761014Z info tf === BEGIN: Building clusters === 2025-06-24T15:46:27.763764Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:46:27.763774Z info tf === DONE: Building clusters === 2025-06-24T15:46:27.763793Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:46:27.763817Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:46:27.763823Z info tf === Istio Component Config === 2025-06-24T15:46:27.763866Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:46:27.763873Z info tf ================================ 2025-06-24T15:46:27.770025Z info tf skipping deployment as specified in the config 2025-06-24T15:46:27.770093Z info tf === SUCCEEDED: Deploy Istio in 6.270959ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:46:27.770102Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:46:27.791103Z info tf === SUCCEEDED: Create namespace echo in 20.984323ms === 2025-06-24T15:46:27.827876Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:46:28.184566Z info klog Waiting for caches to sync for echo 2025-06-24T15:46:28.184734Z info klog Waiting for caches to sync for echo 2025-06-24T15:46:28.285703Z info klog Caches are synced for echo 2025-06-24T15:46:28.285781Z info klog Caches are synced for echo 2025-06-24T15:46:39.457587Z info tf === SUCCEEDED: Deploy echo instances in 11.629707442s === 2025-06-24T15:46:39.554209Z info tf Checking pods ready... 2025-06-24T15:46:39.554417Z info tf Checking pods ready... 2025-06-24T15:46:39.563857Z info tf [ 0] zipkin-78496844d5-4sjl5 Pending (Pending) 2025-06-24T15:46:39.764597Z info tf Checking pods ready... 2025-06-24T15:46:39.764689Z info tf Checking pods ready... 2025-06-24T15:46:39.769149Z info tf [ 0] zipkin-78496844d5-4sjl5 Pending (Pending) 2025-06-24T15:46:40.169392Z info tf Checking pods ready... 2025-06-24T15:46:40.169460Z info tf Checking pods ready... 2025-06-24T15:46:40.175889Z info tf [ 0] zipkin-78496844d5-4sjl5 Pending (Pending) 2025-06-24T15:46:40.975942Z info tf Checking pods ready... 2025-06-24T15:46:40.976030Z info tf Checking pods ready... 2025-06-24T15:46:40.980858Z info tf [ 0] zipkin-78496844d5-4sjl5 Pending (Pending) 2025-06-24T15:46:42.581826Z info tf Checking pods ready... 2025-06-24T15:46:42.581934Z info tf Checking pods ready... 2025-06-24T15:46:42.587106Z info tf [ 0] zipkin-78496844d5-4sjl5 Running (container not ready: 'zipkin') 2025-06-24T15:46:45.788121Z info tf Checking pods ready... 2025-06-24T15:46:45.788217Z info tf Checking pods ready... 2025-06-24T15:46:45.792753Z info tf [ 0] zipkin-78496844d5-4sjl5 Running (container not ready: 'zipkin') 2025-06-24T15:46:48.993595Z info tf Checking pods ready... 2025-06-24T15:46:48.993659Z info tf Checking pods ready... 2025-06-24T15:46:48.998046Z info tf [ 0] zipkin-78496844d5-4sjl5 Running (container not ready: 'zipkin') 2025-06-24T15:46:52.198919Z info tf Checking pods ready... 2025-06-24T15:46:52.198971Z info tf Checking pods ready... 2025-06-24T15:46:52.204536Z info tf [ 0] zipkin-78496844d5-4sjl5 Running (Ready) 2025-06-24T15:46:52.342728Z info tf Checking pods ready... 2025-06-24T15:46:52.342805Z info tf Checking pods ready... 2025-06-24T15:46:52.355851Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T15:46:52.556577Z info tf Checking pods ready... 2025-06-24T15:46:52.556650Z info tf Checking pods ready... 2025-06-24T15:46:52.562752Z info tf [ 0] opentelemetry-collector-69596cbbd6-chjvz Pending (Pending) 2025-06-24T15:46:52.963472Z info tf Checking pods ready... 2025-06-24T15:46:52.963579Z info tf Checking pods ready... 2025-06-24T15:46:52.969211Z info tf [ 0] opentelemetry-collector-69596cbbd6-chjvz Pending (Pending) 2025-06-24T15:46:53.769946Z info tf Checking pods ready... 2025-06-24T15:46:53.770014Z info tf Checking pods ready... 2025-06-24T15:46:53.784456Z info tf [ 0] opentelemetry-collector-69596cbbd6-chjvz Pending (Pending) 2025-06-24T15:46:55.385524Z info tf Checking pods ready... 2025-06-24T15:46:55.385600Z info tf Checking pods ready... 2025-06-24T15:46:55.390307Z info tf [ 0] opentelemetry-collector-69596cbbd6-chjvz Running (Ready) 2025-06-24T15:46:55.434353Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (27.670546603s) === 2025-06-24T15:46:55.434403Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 2025-06-24T15:46:55.444042Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:49:25.444772Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.00072479s)' === 2025-06-24T15:49:25.444860Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:49:25.444845Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:49:26.668821Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26e581777b-0e23-4f3b-a982-e8e03b3b6ca1 2025-06-24T15:49:26.669187Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.2240s 2025-06-24T15:49:26.669666Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:26.964264Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26e85e0e3d-7c2b-44ea-84cd-2189aa095511 2025-06-24T15:49:26.964750Z info tf istioctl ([x internal-debug --all configz]): completed after 1.5195s 2025-06-24T15:49:26.965172Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:27.078869Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26f9dbe1fd-e96d-45ed-b2c6-7aa7951ca3f7 2025-06-24T15:49:27.079127Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.6338s 2025-06-24T15:49:27.079819Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (151.79s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (re-run 1) (151.79s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter 2025-06-24T15:46:55.434906Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T15:49:27.234869Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m31.799954668s)' === 2025-06-24T15:49:27.235004Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:49:27.235001Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:49:27.398161Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26705f23f6-4e44-47af-bdb8-f297ae4f93e3 2025-06-24T15:49:27.398524Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1633s 2025-06-24T15:49:27.399130Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:27.604917Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26de6a1fff-dac3-4d28-b9c8-108764df4724 2025-06-24T15:49:27.605125Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3699s 2025-06-24T15:49:27.605900Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:27.708780Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26d1244b82-7d75-45a7-8a64-38cda3dc066e 2025-06-24T15:49:27.708953Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4737s 2025-06-24T15:49:27.709655Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter (152.71s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter (re-run 1) (152.71s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T15:46:55.434786Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:47:45.463224Z error port forward failed: lost connection to pod 2025-06-24T15:47:45.843316Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward terminated 2025-06-24T15:47:45.881526Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:47:52.230403Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:48:51.887443Z error port forward failed: lost connection to pod 2025-06-24T15:48:52.342394Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward terminated 2025-06-24T15:48:52.392561Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:49:28.148200Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m32.713406772s)' === 2025-06-24T15:49:28.148336Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:49:28.148325Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:49:28.463346Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/2761e5df7a-d553-447c-a718-a86f01446545 2025-06-24T15:49:28.463630Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3150s 2025-06-24T15:49:28.464175Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:28.545419Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/27cfde0e8e-f146-4f56-8ea3-ff9644c60344 2025-06-24T15:49:28.545726Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3971s 2025-06-24T15:49:28.546399Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:28.628963Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/27595e3772-86d6-40d9-97bd-c77fc7c47e09 2025-06-24T15:49:28.629124Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4805s 2025-06-24T15:49:28.629805Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (153.86s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 1) (153.86s) 2025-06-24T15:49:29.294637Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T15:49:29.294706Z info tf === Suite "telemetry_tracing_otelcollector" run time: 3m1.530915541s === 2025-06-24T15:49:29.294728Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T15:49:29.294989Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:49:29.294989Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:49:29.453159Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/27b9bc48b0-bef8-4d31-9f2a-76b0952d963a 2025-06-24T15:49:29.453484Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1582s 2025-06-24T15:49:29.454224Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:29.559922Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/27a20c925a-b2f7-422a-a6cf-26b4ccd5646a 2025-06-24T15:49:29.560119Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2648s 2025-06-24T15:49:29.560947Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:29.634442Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/27ea26b116-3fea-434c-8450-62f5ab64b976 2025-06-24T15:49:29.634593Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3393s 2025-06-24T15:49:29.635154Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:30.465710Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:49:30.465892Z info tf === SUCCEEDED: Cleanup Istio in 157.915µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3543231 2025-06-24T15:49:42.179190Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:49:42.179238Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:49:42.210928Z info tf === Test Framework Settings === 2025-06-24T15:49:42.211026Z info tf TestID: telemetry_tracing_otelcollector RunID: ec95a98d-5ab9-44ef-941f-a4ad9b41494d NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:49:42.211034Z info tf =============================== 2025-06-24T15:49:42.211066Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:49:42.211108Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:49:42.211123Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:49:42.211143Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:49:42.211154Z info tf === BEGIN: Building clusters === 2025-06-24T15:49:42.213539Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:49:42.213548Z info tf === DONE: Building clusters === 2025-06-24T15:49:42.213563Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:49:42.213582Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:49:42.213587Z info tf === Istio Component Config === 2025-06-24T15:49:42.213620Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:49:42.213624Z info tf ================================ 2025-06-24T15:49:42.219182Z info tf skipping deployment as specified in the config 2025-06-24T15:49:42.219252Z info tf === SUCCEEDED: Deploy Istio in 5.66468ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:49:42.219260Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:49:42.234881Z info tf === SUCCEEDED: Create namespace echo in 15.603812ms === 2025-06-24T15:49:42.272424Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:49:42.278153Z error tf === FAILED: Deploy echo instances === 2025-06-24T15:49:42.278176Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1897964835.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:49:42.278213Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:49:42.278227Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:49:42.395857Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:49:42.408008Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:49:42.460554Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:39Z/2925de4408-f37b-4b1b-b6f2-caec85661f3d 2025-06-24T15:49:42.460834Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1823s 2025-06-24T15:49:42.461521Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:42.549836Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:39Z/2915a509e1-d527-419e-82fa-6c8ecd2f3736 2025-06-24T15:49:42.549956Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2714s 2025-06-24T15:49:42.550667Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:42.632557Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:39Z/29be213b8c-61bd-4307-910d-e0e1caf696b3 2025-06-24T15:49:42.632678Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3541s 2025-06-24T15:49:42.633231Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:42.655688Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1897964835.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:49:42.655727Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1897964835.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T15:49:42.655733Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.1897964835.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:49:42.655757Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:49:42.655793Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:49:42.763365Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:49:42.765489Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:49:42.774833Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:39Z/2926b2d270-a958-4353-a096-b746e338af1d 2025-06-24T15:49:42.774950Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1189s 2025-06-24T15:49:42.775489Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:42.878345Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:39Z/295c46dec9-a414-47cf-9f26-0a410c079a96 2025-06-24T15:49:42.878472Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2224s 2025-06-24T15:49:42.879042Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:43.035829Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:39Z/295a3daca9-9ef2-4821-96e6-458f6f712361 2025-06-24T15:49:43.036041Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3799s 2025-06-24T15:49:43.036523Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:43.068495Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:49:43.068527Z info tf === SUCCEEDED: Cleanup Istio in 27.786µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3544614 2025-06-24T15:49:53.466039Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:49:53.466091Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:49:53.501054Z info tf === Test Framework Settings === 2025-06-24T15:49:53.501303Z info tf TestID: telemetry_tracing_otelcollector RunID: 18c11ad7-cf17-444c-ba1f-791dfe02fa37 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:49:53.501313Z info tf =============================== 2025-06-24T15:49:53.501362Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:49:53.501457Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:49:53.501481Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:49:53.501512Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:49:53.501530Z info tf === BEGIN: Building clusters === 2025-06-24T15:49:53.506834Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:49:53.506848Z info tf === DONE: Building clusters === 2025-06-24T15:49:53.506887Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:49:53.506926Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:49:53.506938Z info tf === Istio Component Config === 2025-06-24T15:49:53.507010Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:49:53.507019Z info tf ================================ 2025-06-24T15:49:53.512295Z info tf skipping deployment as specified in the config 2025-06-24T15:49:53.512315Z info tf === SUCCEEDED: Deploy Istio in 5.38392ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:49:53.512323Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:49:53.536728Z info tf === SUCCEEDED: Create namespace echo in 24.38517ms === 2025-06-24T15:49:53.619333Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:49:53.897831Z info klog Waiting for caches to sync for echo 2025-06-24T15:49:53.899527Z info klog Waiting for caches to sync for echo 2025-06-24T15:49:53.998854Z info klog Caches are synced for echo 2025-06-24T15:49:54.000405Z info klog Caches are synced for echo 2025-06-24T15:50:03.504537Z info tf === SUCCEEDED: Deploy echo instances in 9.885185774s === 2025-06-24T15:50:03.564210Z info tf Checking pods ready... 2025-06-24T15:50:03.564240Z info tf Checking pods ready... 2025-06-24T15:50:03.596683Z info tf [ 0] zipkin-78496844d5-m69hr Pending (Pending) 2025-06-24T15:50:03.797486Z info tf Checking pods ready... 2025-06-24T15:50:03.797648Z info tf Checking pods ready... 2025-06-24T15:50:03.802666Z info tf [ 0] zipkin-78496844d5-m69hr Pending (Pending) 2025-06-24T15:50:04.202843Z info tf Checking pods ready... 2025-06-24T15:50:04.202955Z info tf Checking pods ready... 2025-06-24T15:50:04.207715Z info tf [ 0] zipkin-78496844d5-m69hr Pending (Pending) 2025-06-24T15:50:05.008267Z info tf Checking pods ready... 2025-06-24T15:50:05.008314Z info tf Checking pods ready... 2025-06-24T15:50:05.012887Z info tf [ 0] zipkin-78496844d5-m69hr Pending (Pending) 2025-06-24T15:50:06.613883Z info tf Checking pods ready... 2025-06-24T15:50:06.613951Z info tf Checking pods ready... 2025-06-24T15:50:06.619235Z info tf [ 0] zipkin-78496844d5-m69hr Running (container not ready: 'zipkin') 2025-06-24T15:50:09.820023Z info tf Checking pods ready... 2025-06-24T15:50:09.820098Z info tf Checking pods ready... 2025-06-24T15:50:09.826085Z info tf [ 0] zipkin-78496844d5-m69hr Running (container not ready: 'zipkin') 2025-06-24T15:50:13.027231Z info tf Checking pods ready... 2025-06-24T15:50:13.027287Z info tf Checking pods ready... 2025-06-24T15:50:13.031842Z info tf [ 0] zipkin-78496844d5-m69hr Running (container not ready: 'zipkin') 2025-06-24T15:50:16.232962Z info tf Checking pods ready... 2025-06-24T15:50:16.233026Z info tf Checking pods ready... 2025-06-24T15:50:16.237956Z info tf [ 0] zipkin-78496844d5-m69hr Running (Ready) 2025-06-24T15:50:16.388230Z info tf Checking pods ready... 2025-06-24T15:50:16.388278Z info tf Checking pods ready... 2025-06-24T15:50:16.397769Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T15:50:16.597890Z info tf Checking pods ready... 2025-06-24T15:50:16.597986Z info tf Checking pods ready... 2025-06-24T15:50:16.609931Z info tf [ 0] opentelemetry-collector-69596cbbd6-cpgvv Pending (Pending) 2025-06-24T15:50:17.010848Z info tf Checking pods ready... 2025-06-24T15:50:17.010927Z info tf Checking pods ready... 2025-06-24T15:50:17.015274Z info tf [ 0] opentelemetry-collector-69596cbbd6-cpgvv Pending (Pending) 2025-06-24T15:50:17.815408Z info tf Checking pods ready... 2025-06-24T15:50:17.815483Z info tf Checking pods ready... 2025-06-24T15:50:17.821871Z info tf [ 0] opentelemetry-collector-69596cbbd6-cpgvv Pending (Pending) 2025-06-24T15:50:19.422913Z info tf Checking pods ready... 2025-06-24T15:50:19.422985Z info tf Checking pods ready... 2025-06-24T15:50:19.428076Z info tf [ 0] opentelemetry-collector-69596cbbd6-cpgvv Running (Ready) 2025-06-24T15:50:19.467425Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (25.960523429s) === 2025-06-24T15:50:19.467461Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 2025-06-24T15:50:19.478869Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:52:49.479741Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000863306s)' === 2025-06-24T15:52:49.479892Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:52:49.479864Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:52:49.620226Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/3742dceb6f-b1a4-4fb8-b239-5bed0d27183d 2025-06-24T15:52:49.620504Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1399s 2025-06-24T15:52:49.620955Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:49.722291Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/3702bd74c5-ff05-4abf-908d-508405a261c3 2025-06-24T15:52:49.722622Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2420s 2025-06-24T15:52:49.722968Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:49.848277Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/377532dcc3-3294-4ecc-a859-f972512763ba 2025-06-24T15:52:49.848471Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3679s 2025-06-24T15:52:49.848967Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (150.75s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 1) (150.75s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter 2025-06-24T15:50:19.467860Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T15:52:50.226797Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.758927532s)' === 2025-06-24T15:52:50.226909Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:52:50.226940Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:52:50.338306Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/377ba1f95f-7f80-4070-9d38-ae148dacae71 2025-06-24T15:52:50.338453Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1112s 2025-06-24T15:52:50.339008Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:50.423146Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/3707b1a05f-89eb-4d35-9360-b3e0b2c99ed6 2025-06-24T15:52:50.423374Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.1961s 2025-06-24T15:52:50.423869Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:50.517582Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/37db79d218-1b82-4693-9a23-2a985c147903 2025-06-24T15:52:50.518635Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2913s 2025-06-24T15:52:50.519427Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter (151.51s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 1) (151.51s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T15:50:19.467732Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:51:08.200205Z error port forward failed: lost connection to pod 2025-06-24T15:51:08.545421Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward terminated 2025-06-24T15:51:08.592668Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:51:16.270210Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:52:14.598344Z error port forward failed: lost connection to pod 2025-06-24T15:52:15.045278Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward terminated 2025-06-24T15:52:15.081011Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:52:50.973643Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.505900602s)' === 2025-06-24T15:52:50.973795Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:52:50.973784Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:52:51.127432Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/38d732b9c5-60a2-474c-afc2-aec8b68cc50c 2025-06-24T15:52:51.127582Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1533s 2025-06-24T15:52:51.128214Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:51.221994Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/389f96d9aa-ddfe-40cb-9ac8-3c19462ec779 2025-06-24T15:52:51.222287Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2480s 2025-06-24T15:52:51.222804Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:51.325782Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/38d46cc54b-9011-4101-8ed2-49c1f256f2ba 2025-06-24T15:52:51.325908Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3516s 2025-06-24T15:52:51.326484Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (152.75s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 1) (152.75s) 2025-06-24T15:52:52.221209Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T15:52:52.221307Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m58.714435091s === 2025-06-24T15:52:52.221325Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T15:52:52.221582Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:52:52.221596Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:52:52.374762Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/384d26df47-af6b-420b-9b91-7078ff10d148 2025-06-24T15:52:52.375053Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1532s 2025-06-24T15:52:52.375738Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:52.488005Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/386f15e4a0-d211-4740-a7a2-feb4d041db7f 2025-06-24T15:52:52.488164Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2663s 2025-06-24T15:52:52.488992Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:52.586307Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/383a590d75-c04f-423e-b7ad-00b8723629d7 2025-06-24T15:52:52.586520Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3646s 2025-06-24T15:52:52.587211Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:53.307225Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:52:53.307363Z info tf === SUCCEEDED: Cleanup Istio in 120.204µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3546138 2025-06-24T15:53:04.316588Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:53:04.316642Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:53:04.346823Z info tf === Test Framework Settings === 2025-06-24T15:53:04.346901Z info tf TestID: telemetry_tracing_otelcollector RunID: 9f8cee56-aa10-417f-b1db-9682f6421773 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:53:04.346907Z info tf =============================== 2025-06-24T15:53:04.346935Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:53:04.346972Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:53:04.346981Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:53:04.347004Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:53:04.347015Z info tf === BEGIN: Building clusters === 2025-06-24T15:53:04.350169Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:53:04.350204Z info tf === DONE: Building clusters === 2025-06-24T15:53:04.350230Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:53:04.350259Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:53:04.350264Z info tf === Istio Component Config === 2025-06-24T15:53:04.350336Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:53:04.350344Z info tf ================================ 2025-06-24T15:53:04.353982Z info tf skipping deployment as specified in the config 2025-06-24T15:53:04.353998Z info tf === SUCCEEDED: Deploy Istio in 3.735956ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:53:04.354004Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:53:04.373580Z info tf === SUCCEEDED: Create namespace echo in 19.557034ms === 2025-06-24T15:53:04.405185Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:53:04.412074Z error tf === FAILED: Deploy echo instances === 2025-06-24T15:53:04.412095Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3021149345.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:53:04.412132Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:53:04.412130Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:53:04.490674Z error klog an error occurred forwarding 38127 -> 15020: error forwarding port 15020 to pod d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823, uid : port forward into network namespace "/var/run/netns/38bc8050-20ba-48e4-98f7-cd85efd77ba2": failed to connect to localhost:15020 inside namespace d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.492181Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:38127/debug/ndsz": EOF 2025-06-24T15:53:04.492233Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.494109Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:38127/stats/prometheus": dial tcp 127.0.0.1:38127: connect: connection refused 2025-06-24T15:53:04.513543Z error klog an error occurred forwarding 44669 -> 15000: error forwarding port 15000 to pod d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823, uid : port forward into network namespace "/var/run/netns/38bc8050-20ba-48e4-98f7-cd85efd77ba2": failed to connect to localhost:15000 inside namespace d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.515481Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.515479Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:44669/config_dump?include_eds=true": EOF 2025-06-24T15:53:04.516253Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:44669/clusters": dial tcp 127.0.0.1:44669: connect: connection refused 2025-06-24T15:53:04.530836Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:00Z/41f22bd8cd-2d7e-414f-a2e1-d1b39d9626bf 2025-06-24T15:53:04.531023Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1186s 2025-06-24T15:53:04.531651Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:53:04.581386Z error klog an error occurred forwarding 43859 -> 15020: error forwarding port 15020 to pod 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf, uid : port forward into network namespace "/var/run/netns/74c4959b-71bc-452a-8996-60442ebf3a80": failed to connect to localhost:15020 inside namespace 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.581478Z error klog an error occurred forwarding 39149 -> 15000: error forwarding port 15000 to pod 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf, uid : port forward into network namespace "/var/run/netns/74c4959b-71bc-452a-8996-60442ebf3a80": failed to connect to localhost:15000 inside namespace 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.582928Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:43859/debug/ndsz": EOF 2025-06-24T15:53:04.582968Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.583076Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:39149/config_dump?include_eds=true": EOF 2025-06-24T15:53:04.583098Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.583668Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:39149/clusters": dial tcp 127.0.0.1:39149: connect: connection refused 2025-06-24T15:53:04.583679Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:43859/stats/prometheus": dial tcp 127.0.0.1:43859: connect: connection refused 2025-06-24T15:53:04.612886Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:00Z/41c47ac126-05fd-4ba4-8758-4e46356ce814 2025-06-24T15:53:04.612970Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2005s 2025-06-24T15:53:04.613598Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:53:04.743462Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:00Z/4122954b3c-3be7-4665-a46e-969bec5835ff 2025-06-24T15:53:04.743597Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3311s 2025-06-24T15:53:04.744356Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:53:04.794817Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3021149345.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:53:04.794874Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3021149345.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T15:53:04.794881Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.3021149345.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:53:04.794910Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:53:04.794935Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:53:04.877791Z error klog an error occurred forwarding 35489 -> 15000: error forwarding port 15000 to pod 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf, uid : port forward into network namespace "/var/run/netns/74c4959b-71bc-452a-8996-60442ebf3a80": failed to connect to localhost:15000 inside namespace 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.879697Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.879740Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:35489/config_dump?include_eds=true": EOF 2025-06-24T15:53:04.880576Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:35489/clusters": dial tcp 127.0.0.1:35489: connect: connection refused 2025-06-24T15:53:04.883033Z error klog an error occurred forwarding 43497 -> 15000: error forwarding port 15000 to pod d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823, uid : port forward into network namespace "/var/run/netns/38bc8050-20ba-48e4-98f7-cd85efd77ba2": failed to connect to localhost:15000 inside namespace d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.885190Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.885235Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:43497/config_dump?include_eds=true": EOF 2025-06-24T15:53:04.886224Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:43497/clusters": dial tcp 127.0.0.1:43497: connect: connection refused 2025-06-24T15:53:04.890770Z error klog an error occurred forwarding 41961 -> 15020: error forwarding port 15020 to pod d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823, uid : port forward into network namespace "/var/run/netns/38bc8050-20ba-48e4-98f7-cd85efd77ba2": failed to connect to localhost:15020 inside namespace d895c01fc025dded715e42f11ccaecc7d44b8e3c80eea88819aa72d334b5b823: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.892443Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:41961/debug/ndsz": EOF 2025-06-24T15:53:04.892501Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.893216Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-nvrw5 for: Get "http://localhost:41961/stats/prometheus": dial tcp 127.0.0.1:41961: connect: connection refused 2025-06-24T15:53:04.906862Z error klog an error occurred forwarding 34607 -> 15020: error forwarding port 15020 to pod 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf, uid : port forward into network namespace "/var/run/netns/74c4959b-71bc-452a-8996-60442ebf3a80": failed to connect to localhost:15020 inside namespace 71c7638455e4fb4728e00b92cfa1a2e40507e4a10d0df1714ec7755c65c180bf: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:53:04.908527Z error port forward failed: lost connection to pod 2025-06-24T15:53:04.908534Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:34607/debug/ndsz": EOF 2025-06-24T15:53:04.909439Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-jjvzz for: Get "http://localhost:34607/stats/prometheus": dial tcp 127.0.0.1:34607: connect: connection refused 2025-06-24T15:53:04.937226Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:00Z/412ff7976e-f9c5-4483-a66a-328089eafafb 2025-06-24T15:53:04.937467Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1421s 2025-06-24T15:53:04.938040Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:53:05.050882Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:00Z/413c5bbf23-eb88-48a8-85f6-9e966cc2b86a 2025-06-24T15:53:05.050989Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2556s 2025-06-24T15:53:05.051731Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:53:05.169214Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:00Z/41ed29d591-235b-4a22-92dd-525a52bb3d16 2025-06-24T15:53:05.169442Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3741s 2025-06-24T15:53:05.170066Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:53:05.205900Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:53:05.205964Z info tf === SUCCEEDED: Cleanup Istio in 60.897µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3547375 2025-06-24T15:53:16.307917Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:53:16.307988Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:53:16.340804Z info tf === Test Framework Settings === 2025-06-24T15:53:16.340881Z info tf TestID: telemetry_tracing_otelcollector RunID: bfa70565-e653-4853-8258-ca064081e09a NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:53:16.340886Z info tf =============================== 2025-06-24T15:53:16.340918Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:53:16.340951Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:53:16.340962Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:53:16.340979Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:53:16.340989Z info tf === BEGIN: Building clusters === 2025-06-24T15:53:16.343540Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:53:16.343548Z info tf === DONE: Building clusters === 2025-06-24T15:53:16.343565Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:53:16.343587Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:53:16.343593Z info tf === Istio Component Config === 2025-06-24T15:53:16.343622Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:53:16.343627Z info tf ================================ 2025-06-24T15:53:16.346878Z info tf skipping deployment as specified in the config 2025-06-24T15:53:16.346893Z info tf === SUCCEEDED: Deploy Istio in 3.301487ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:53:16.346900Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:53:16.380327Z info tf === SUCCEEDED: Create namespace echo in 33.407414ms === 2025-06-24T15:53:16.518657Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:53:16.845655Z info klog Waiting for caches to sync for echo 2025-06-24T15:53:16.932377Z info klog Waiting for caches to sync for echo 2025-06-24T15:53:16.946226Z info klog Caches are synced for echo 2025-06-24T15:53:17.033630Z info klog Caches are synced for echo 2025-06-24T15:53:26.539736Z info tf === SUCCEEDED: Deploy echo instances in 10.021014891s === 2025-06-24T15:53:26.664579Z info tf Checking pods ready... 2025-06-24T15:53:26.664620Z info tf Checking pods ready... 2025-06-24T15:53:26.682850Z info tf [ 0] zipkin-78496844d5-z8rq6 Pending (Pending) 2025-06-24T15:53:26.883732Z info tf Checking pods ready... 2025-06-24T15:53:26.883841Z info tf Checking pods ready... 2025-06-24T15:53:26.891547Z info tf [ 0] zipkin-78496844d5-z8rq6 Pending (Pending) 2025-06-24T15:53:27.291931Z info tf Checking pods ready... 2025-06-24T15:53:27.292039Z info tf Checking pods ready... 2025-06-24T15:53:27.300784Z info tf [ 0] zipkin-78496844d5-z8rq6 Pending (Pending) 2025-06-24T15:53:28.101312Z info tf Checking pods ready... 2025-06-24T15:53:28.101377Z info tf Checking pods ready... 2025-06-24T15:53:28.106793Z info tf [ 0] zipkin-78496844d5-z8rq6 Pending (Pending) 2025-06-24T15:53:29.707718Z info tf Checking pods ready... 2025-06-24T15:53:29.707769Z info tf Checking pods ready... 2025-06-24T15:53:29.714950Z info tf [ 0] zipkin-78496844d5-z8rq6 Running (container not ready: 'zipkin') 2025-06-24T15:53:32.915827Z info tf Checking pods ready... 2025-06-24T15:53:32.915877Z info tf Checking pods ready... 2025-06-24T15:53:32.922314Z info tf [ 0] zipkin-78496844d5-z8rq6 Running (container not ready: 'zipkin') 2025-06-24T15:53:36.122809Z info tf Checking pods ready... 2025-06-24T15:53:36.122872Z info tf Checking pods ready... 2025-06-24T15:53:36.127643Z info tf [ 0] zipkin-78496844d5-z8rq6 Running (container not ready: 'zipkin') 2025-06-24T15:53:39.327774Z info tf Checking pods ready... 2025-06-24T15:53:39.327853Z info tf Checking pods ready... 2025-06-24T15:53:39.334127Z info tf [ 0] zipkin-78496844d5-z8rq6 Running (Ready) 2025-06-24T15:53:39.472844Z info tf Checking pods ready... 2025-06-24T15:53:39.472885Z info tf Checking pods ready... 2025-06-24T15:53:39.477761Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T15:53:39.678590Z info tf Checking pods ready... 2025-06-24T15:53:39.678652Z info tf Checking pods ready... 2025-06-24T15:53:39.683359Z info tf [ 0] opentelemetry-collector-69596cbbd6-sdfdx Pending (Pending) 2025-06-24T15:53:40.083711Z info tf Checking pods ready... 2025-06-24T15:53:40.083787Z info tf Checking pods ready... 2025-06-24T15:53:40.088138Z info tf [ 0] opentelemetry-collector-69596cbbd6-sdfdx Pending (Pending) 2025-06-24T15:53:40.888961Z info tf Checking pods ready... 2025-06-24T15:53:40.889038Z info tf Checking pods ready... 2025-06-24T15:53:40.894532Z info tf [ 0] opentelemetry-collector-69596cbbd6-sdfdx Pending (Pending) 2025-06-24T15:53:42.494973Z info tf Checking pods ready... 2025-06-24T15:53:42.495051Z info tf Checking pods ready... 2025-06-24T15:53:42.499838Z info tf [ 0] opentelemetry-collector-69596cbbd6-sdfdx Running (Ready) 2025-06-24T15:53:42.544664Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (26.20108811s) === 2025-06-24T15:53:42.544703Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 2025-06-24T15:53:42.554520Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:56:12.555845Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.001317097s)' === 2025-06-24T15:56:12.555961Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:56:12.555983Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:56:12.822215Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/496a49b029-3545-4ef5-9582-a21be7b4612e 2025-06-24T15:56:12.822543Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2662s 2025-06-24T15:56:12.822951Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:12.918124Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/4924a48e35-0b90-42ad-b18a-638038e64542 2025-06-24T15:56:12.918319Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3620s 2025-06-24T15:56:12.918853Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:13.003841Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/49095eeb08-c512-4095-b3f1-6f8ab2738b66 2025-06-24T15:56:13.003974Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4476s 2025-06-24T15:56:13.004534Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (150.85s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (re-run 1) (150.85s) === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors 2025-06-24T15:53:42.545124Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T15:56:13.400543Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m30.855409295s)' === 2025-06-24T15:56:13.400688Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:56:13.400708Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:56:13.786272Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/495c793321-a1a3-411e-a1fd-e4f821bdfec1 2025-06-24T15:56:13.786597Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3856s 2025-06-24T15:56:13.787214Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:13.861288Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/49d1e96c4f-ba30-47db-94d7-d7ab6bcf6238 2025-06-24T15:56:13.861501Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4605s 2025-06-24T15:56:13.862388Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:13.941838Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/493536e542-2b26-4f9c-a589-80ad03678431 2025-06-24T15:56:13.942063Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5411s 2025-06-24T15:56:13.942623Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors (151.78s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors (re-run 1) (151.78s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T15:53:42.544990Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:54:31.274449Z error port forward failed: lost connection to pod 2025-06-24T15:54:31.544590Z warn pod: echo/server-v1-6f45fcb457-wmmqg port forward terminated 2025-06-24T15:54:31.587086Z warn pod: echo/server-v1-6f45fcb457-wmmqg port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:54:39.359628Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:55:37.592106Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:56:14.327572Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.782577971s)' === 2025-06-24T15:56:14.327660Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:56:14.327659Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:56:14.487637Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/50a739f71e-a38a-4c82-840e-308d46a0bd91 2025-06-24T15:56:14.487941Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1600s 2025-06-24T15:56:14.488476Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:14.590219Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/506fef122f-50fd-4c6c-8fd9-1b75fc0d572b 2025-06-24T15:56:14.590395Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2625s 2025-06-24T15:56:14.591239Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:14.672211Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/50f4050e77-a30b-448c-a152-1bc23b6d3cea 2025-06-24T15:56:14.672422Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3445s 2025-06-24T15:56:14.673008Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (153.00s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 1) (153.00s) 2025-06-24T15:56:15.540433Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T15:56:15.540485Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m59.196924836s === 2025-06-24T15:56:15.540498Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T15:56:15.540743Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:56:15.540748Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:56:15.716011Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/50dc9c009d-53d7-41b5-9058-6010a6447391 2025-06-24T15:56:15.716336Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1753s 2025-06-24T15:56:15.717031Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:15.822343Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/50cc656faa-4caf-451f-8bce-47994d6f2c6f 2025-06-24T15:56:15.822525Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2815s 2025-06-24T15:56:15.823323Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:15.947415Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/50e022fd5f-a660-48ef-8069-699f4f6c86e4 2025-06-24T15:56:15.947710Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4066s 2025-06-24T15:56:15.948265Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:16.799483Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:56:16.799613Z info tf === SUCCEEDED: Cleanup Istio in 113.938µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3548896 2025-06-24T15:56:27.539141Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:56:27.539198Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:56:27.572265Z info tf === Test Framework Settings === 2025-06-24T15:56:27.572362Z info tf TestID: telemetry_tracing_otelcollector RunID: 248e22ee-9e4b-4b6f-9421-cc46176d6ecd NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:56:27.572367Z info tf =============================== 2025-06-24T15:56:27.572393Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:56:27.572435Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:56:27.572446Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:56:27.572462Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:56:27.572469Z info tf === BEGIN: Building clusters === 2025-06-24T15:56:27.574986Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:56:27.574995Z info tf === DONE: Building clusters === 2025-06-24T15:56:27.575013Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:56:27.575034Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:56:27.575040Z info tf === Istio Component Config === 2025-06-24T15:56:27.575075Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:56:27.575079Z info tf ================================ 2025-06-24T15:56:27.580475Z info tf skipping deployment as specified in the config 2025-06-24T15:56:27.580539Z info tf === SUCCEEDED: Deploy Istio in 5.498447ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:56:27.580548Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:56:27.594262Z info tf === SUCCEEDED: Create namespace echo in 13.707024ms === 2025-06-24T15:56:27.648004Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:56:27.659674Z error tf === FAILED: Deploy echo instances === 2025-06-24T15:56:27.659704Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.176041726.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:56:27.659761Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:56:27.659774Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:56:27.936645Z error klog an error occurred forwarding 37111 -> 15020: error forwarding port 15020 to pod 4186ce6069615509e45df71d6e5a3ea83b209ef1ac2d86f2002308356621650c, uid : port forward into network namespace "/var/run/netns/67eccd30-e3c8-4722-8b83-4ae010a93b2c": failed to connect to localhost:15020 inside namespace 4186ce6069615509e45df71d6e5a3ea83b209ef1ac2d86f2002308356621650c: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:56:27.938897Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-wmmqg for: Get "http://localhost:37111/debug/ndsz": EOF 2025-06-24T15:56:27.938961Z error port forward failed: lost connection to pod 2025-06-24T15:56:27.939921Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-wmmqg for: Get "http://localhost:37111/stats/prometheus": dial tcp 127.0.0.1:37111: connect: connection refused 2025-06-24T15:56:27.979183Z error klog an error occurred forwarding 40001 -> 15020: error forwarding port 15020 to pod ed65168c6e271de9c7a3575de12f1782e3add5cf919dde6be4cc5dfed6ae7582, uid : port forward into network namespace "/var/run/netns/0346a66a-41b4-4231-ad52-189613be0779": failed to connect to localhost:15020 inside namespace ed65168c6e271de9c7a3575de12f1782e3add5cf919dde6be4cc5dfed6ae7582: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:56:27.980213Z error klog an error occurred forwarding 35459 -> 15000: error forwarding port 15000 to pod ed65168c6e271de9c7a3575de12f1782e3add5cf919dde6be4cc5dfed6ae7582, uid : port forward into network namespace "/var/run/netns/0346a66a-41b4-4231-ad52-189613be0779": failed to connect to localhost:15000 inside namespace ed65168c6e271de9c7a3575de12f1782e3add5cf919dde6be4cc5dfed6ae7582: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:56:27.981007Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-skb9w for: Get "http://localhost:40001/debug/ndsz": EOF 2025-06-24T15:56:27.981154Z error port forward failed: lost connection to pod 2025-06-24T15:56:27.981935Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-skb9w for: Get "http://localhost:35459/config_dump?include_eds=true": EOF 2025-06-24T15:56:27.981951Z error port forward failed: lost connection to pod 2025-06-24T15:56:27.982062Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-skb9w for: Get "http://localhost:40001/stats/prometheus": dial tcp 127.0.0.1:40001: connect: connection refused 2025-06-24T15:56:27.982688Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-skb9w for: Get "http://localhost:35459/clusters": dial tcp 127.0.0.1:35459: connect: connection refused 2025-06-24T15:56:28.034749Z error klog an error occurred forwarding 37525 -> 15000: error forwarding port 15000 to pod 4186ce6069615509e45df71d6e5a3ea83b209ef1ac2d86f2002308356621650c, uid : port forward into network namespace "/var/run/netns/67eccd30-e3c8-4722-8b83-4ae010a93b2c": failed to connect to localhost:15000 inside namespace 4186ce6069615509e45df71d6e5a3ea83b209ef1ac2d86f2002308356621650c: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:56:28.036446Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-wmmqg for: Get "http://localhost:37525/config_dump?include_eds=true": EOF 2025-06-24T15:56:28.036459Z error port forward failed: lost connection to pod 2025-06-24T15:56:28.037149Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-wmmqg for: Get "http://localhost:37525/clusters": dial tcp 127.0.0.1:37525: connect: connection refused 2025-06-24T15:56:28.043734Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.176041726.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:56:28.043764Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.176041726.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T15:56:28.043773Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.176041726.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:56:28.043807Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:56:28.043823Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:56:28.360709Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-wmmqg_echo") 2025-06-24T15:56:28.373425Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-6f45fcb457-wmmqg_echo") 2025-06-24T15:56:28.399569Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:24Z/521574a6f7-2faf-4a8d-9d8f-ba3adfccd836 2025-06-24T15:56:28.399872Z info tf istioctl ([x internal-debug --all configz]): completed after 0.7398s 2025-06-24T15:56:28.400400Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:28.432680Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-5c7cd449cf-skb9w_echo") 2025-06-24T15:56:28.433024Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-cluster-0-v1-5c7cd449cf-skb9w_echo") 2025-06-24T15:56:28.500770Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:56:28.500761Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T15:56:28.505127Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:56:28.505152Z info tf === SUCCEEDED: Cleanup Istio in 19.25µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3550311 2025-06-24T15:56:39.208366Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:56:39.208416Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:56:39.238153Z info tf === Test Framework Settings === 2025-06-24T15:56:39.238239Z info tf TestID: telemetry_tracing_otelcollector RunID: 06eb50e6-7252-4f3a-a5e9-e1d4fafbb1cd NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:56:39.238247Z info tf =============================== 2025-06-24T15:56:39.238287Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:56:39.238330Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:56:39.238343Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:56:39.238364Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:56:39.238376Z info tf === BEGIN: Building clusters === 2025-06-24T15:56:39.240887Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:56:39.240896Z info tf === DONE: Building clusters === 2025-06-24T15:56:39.240911Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:56:39.240933Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:56:39.240938Z info tf === Istio Component Config === 2025-06-24T15:56:39.240966Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:56:39.240971Z info tf ================================ 2025-06-24T15:56:39.245880Z info tf skipping deployment as specified in the config 2025-06-24T15:56:39.245952Z info tf === SUCCEEDED: Deploy Istio in 5.013874ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:56:39.245959Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:56:39.268580Z info tf === SUCCEEDED: Create namespace echo in 22.613296ms === 2025-06-24T15:56:39.323084Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:56:39.572389Z info klog Waiting for caches to sync for echo 2025-06-24T15:56:39.629961Z info klog Waiting for caches to sync for echo 2025-06-24T15:56:39.673322Z info klog Caches are synced for echo 2025-06-24T15:56:39.730909Z info klog Caches are synced for echo 2025-06-24T15:56:49.238514Z info tf === SUCCEEDED: Deploy echo instances in 9.915428422s === 2025-06-24T15:56:49.378885Z info tf Checking pods ready... 2025-06-24T15:56:49.378955Z info tf Checking pods ready... 2025-06-24T15:56:49.394023Z info tf [ 0] zipkin-78496844d5-j2wh5 Pending (Pending) 2025-06-24T15:56:49.594697Z info tf Checking pods ready... 2025-06-24T15:56:49.594769Z info tf Checking pods ready... 2025-06-24T15:56:49.600888Z info tf [ 0] zipkin-78496844d5-j2wh5 Pending (Pending) 2025-06-24T15:56:50.001046Z info tf Checking pods ready... 2025-06-24T15:56:50.001120Z info tf Checking pods ready... 2025-06-24T15:56:50.006514Z info tf [ 0] zipkin-78496844d5-j2wh5 Pending (Pending) 2025-06-24T15:56:50.806894Z info tf Checking pods ready... 2025-06-24T15:56:50.806923Z info tf Checking pods ready... 2025-06-24T15:56:50.813328Z info tf [ 0] zipkin-78496844d5-j2wh5 Pending (Pending) 2025-06-24T15:56:52.414269Z info tf Checking pods ready... 2025-06-24T15:56:52.414299Z info tf Checking pods ready... 2025-06-24T15:56:52.420527Z info tf [ 0] zipkin-78496844d5-j2wh5 Running (container not ready: 'zipkin') 2025-06-24T15:56:55.621444Z info tf Checking pods ready... 2025-06-24T15:56:55.621474Z info tf Checking pods ready... 2025-06-24T15:56:55.625420Z info tf [ 0] zipkin-78496844d5-j2wh5 Running (container not ready: 'zipkin') 2025-06-24T15:56:58.825966Z info tf Checking pods ready... 2025-06-24T15:56:58.825998Z info tf Checking pods ready... 2025-06-24T15:56:58.831999Z info tf [ 0] zipkin-78496844d5-j2wh5 Running (container not ready: 'zipkin') 2025-06-24T15:57:02.033026Z info tf Checking pods ready... 2025-06-24T15:57:02.033075Z info tf Checking pods ready... 2025-06-24T15:57:02.039586Z info tf [ 0] zipkin-78496844d5-j2wh5 Running (Ready) 2025-06-24T15:57:02.159810Z info tf Checking pods ready... 2025-06-24T15:57:02.159845Z info tf Checking pods ready... 2025-06-24T15:57:02.174213Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T15:57:02.375329Z info tf Checking pods ready... 2025-06-24T15:57:02.375382Z info tf Checking pods ready... 2025-06-24T15:57:02.380095Z info tf [ 0] opentelemetry-collector-69596cbbd6-qm4j6 Pending (Pending) 2025-06-24T15:57:02.780776Z info tf Checking pods ready... 2025-06-24T15:57:02.780820Z info tf Checking pods ready... 2025-06-24T15:57:02.790737Z info tf [ 0] opentelemetry-collector-69596cbbd6-qm4j6 Pending (Pending) 2025-06-24T15:57:03.591749Z info tf Checking pods ready... 2025-06-24T15:57:03.591830Z info tf Checking pods ready... 2025-06-24T15:57:03.599951Z info tf [ 0] opentelemetry-collector-69596cbbd6-qm4j6 Pending (Pending) 2025-06-24T15:57:05.200559Z info tf Checking pods ready... 2025-06-24T15:57:05.200610Z info tf Checking pods ready... 2025-06-24T15:57:05.204756Z info tf [ 0] opentelemetry-collector-69596cbbd6-qm4j6 Running (Ready) 2025-06-24T15:57:05.246224Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (26.005303034s) === 2025-06-24T15:57:05.246267Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 2025-06-24T15:57:05.258271Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:59:35.259171Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.00089766s)' === 2025-06-24T15:59:35.259323Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:59:35.259317Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:59:35.447975Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/6084ba06e5-7c8f-45cb-a6e2-344212da33e5 2025-06-24T15:59:35.448144Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1885s 2025-06-24T15:59:35.448814Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:35.619921Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/60bc80d092-fc12-449e-a83f-5ce328d26793 2025-06-24T15:59:35.620168Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3605s 2025-06-24T15:59:35.620742Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:35.769538Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/60cc569d9b-057e-4b81-b004-0724f8001987 2025-06-24T15:59:35.769753Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5101s 2025-06-24T15:59:35.770504Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (150.89s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 1) (150.89s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata 2025-06-24T15:57:05.246853Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T15:59:36.143493Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.896628363s)' === 2025-06-24T15:59:36.143639Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:59:36.143641Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:59:36.336299Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/608b36e432-34fd-4eb8-9682-5b1b3b6a3e6e 2025-06-24T15:59:36.336602Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1927s 2025-06-24T15:59:36.337073Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:36.462518Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/602add8c88-57fe-496f-b984-abb99d8186ad 2025-06-24T15:59:36.462856Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3189s 2025-06-24T15:59:36.463659Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:36.599194Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/6067d93c6c-85e8-49a3-a057-e5f6e7a32331 2025-06-24T15:59:36.599552Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4556s 2025-06-24T15:59:36.600351Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (151.77s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 1) (151.77s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T15:57:05.246574Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:57:54.492304Z error port forward failed: lost connection to pod 2025-06-24T15:57:54.785132Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward terminated 2025-06-24T15:57:54.847919Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:58:02.067787Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:59:00.855656Z error port forward failed: lost connection to pod 2025-06-24T15:59:01.284349Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward terminated 2025-06-24T15:59:01.319909Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:59:37.015538Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.768961763s)' === 2025-06-24T15:59:37.015630Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:59:37.015637Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:59:37.185382Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/6138f6130e-c775-4271-baa8-28cf994d44f7 2025-06-24T15:59:37.185629Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1697s 2025-06-24T15:59:37.186287Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:37.303504Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/61dfb526ee-1fe1-46be-8abe-70cad589a044 2025-06-24T15:59:37.303743Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2877s 2025-06-24T15:59:37.304301Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:37.402909Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/618331940c-3889-49c2-82ed-8329e07d1bce 2025-06-24T15:59:37.403044Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3870s 2025-06-24T15:59:37.403607Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (153.09s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 1) (153.09s) 2025-06-24T15:59:38.336359Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T15:59:38.336427Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m59.09551492s === 2025-06-24T15:59:38.336447Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T15:59:38.336773Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:59:38.336772Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:59:38.560369Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/6163aa30ad-1073-4bb9-8605-624792fa59b0 2025-06-24T15:59:38.560658Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2233s 2025-06-24T15:59:38.561254Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:38.641450Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/61d9fe0394-16f2-4660-ad53-dbcb89e85b7f 2025-06-24T15:59:38.641568Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3041s 2025-06-24T15:59:38.642226Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:38.730391Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/61f00a5a45-4489-426c-949c-a4c1aa7a2549 2025-06-24T15:59:38.730565Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3931s 2025-06-24T15:59:38.731230Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:39.467810Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:59:39.467913Z info tf === SUCCEEDED: Cleanup Istio in 92.005µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector DONE 2 runs, 21 tests, 21 failures in 1479.070s exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3551938 2025-06-24T15:59:50.149156Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:59:50.149231Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T15:59:50.180893Z info tf === Test Framework Settings === 2025-06-24T15:59:50.180988Z info tf TestID: telemetry_tracing_otelcollector RunID: a211d6d6-ccbb-47c8-a60b-d716c5fa2a29 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T15:59:50.180993Z info tf =============================== 2025-06-24T15:59:50.181024Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T15:59:50.181074Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T15:59:50.181086Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T15:59:50.181103Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T15:59:50.181111Z info tf === BEGIN: Building clusters === 2025-06-24T15:59:50.187210Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:59:50.187306Z info tf === DONE: Building clusters === 2025-06-24T15:59:50.187344Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T15:59:50.187384Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:59:50.187394Z info tf === Istio Component Config === 2025-06-24T15:59:50.187452Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T15:59:50.187459Z info tf ================================ 2025-06-24T15:59:50.192193Z info tf skipping deployment as specified in the config 2025-06-24T15:59:50.192214Z info tf === SUCCEEDED: Deploy Istio in 4.824447ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T15:59:50.192225Z info tf === BEGIN: Create namespace echo === 2025-06-24T15:59:50.206820Z info tf === SUCCEEDED: Create namespace echo in 14.572151ms === 2025-06-24T15:59:50.248368Z info tf === BEGIN: Deploy echo instances === 2025-06-24T15:59:50.256516Z error tf === FAILED: Deploy echo instances === 2025-06-24T15:59:50.256587Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2211072733.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:59:50.256639Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:59:50.256637Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:59:50.399507Z error klog an error occurred forwarding 38089 -> 15020: error forwarding port 15020 to pod bac434c9af35879f1d8e4b2bba16bf5aa0aee807584c79058bc647b415837175, uid : port forward into network namespace "/var/run/netns/71c2d497-ca45-4dbb-b144-304b49e09198": failed to connect to localhost:15020 inside namespace bac434c9af35879f1d8e4b2bba16bf5aa0aee807584c79058bc647b415837175: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:59:50.401160Z error port forward failed: lost connection to pod 2025-06-24T15:59:50.401232Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-qhds7 for: Get "http://localhost:38089/debug/ndsz": EOF 2025-06-24T15:59:50.401936Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-qhds7 for: Get "http://localhost:38089/stats/prometheus": dial tcp 127.0.0.1:38089: connect: connection refused 2025-06-24T15:59:50.405968Z error klog an error occurred forwarding 41725 -> 15020: error forwarding port 15020 to pod ce94c91cee8a86b967950aca46edbde08f802bd144033d2ecd56d70da9b19d6c, uid : port forward into network namespace "/var/run/netns/aafc720c-d63c-45e5-8e52-be6fe923e609": failed to connect to localhost:15020 inside namespace ce94c91cee8a86b967950aca46edbde08f802bd144033d2ecd56d70da9b19d6c: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:59:50.407579Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-j4msc for: Get "http://localhost:41725/debug/ndsz": EOF 2025-06-24T15:59:50.407643Z error port forward failed: lost connection to pod 2025-06-24T15:59:50.408199Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-j4msc for: Get "http://localhost:41725/stats/prometheus": dial tcp 127.0.0.1:41725: connect: connection refused 2025-06-24T15:59:50.414892Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:45Z/6332817af6-d6e0-4d1f-9b11-603f853b09c6 2025-06-24T15:59:50.415052Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1581s 2025-06-24T15:59:50.415514Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:50.521739Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:45Z/63a2ce09f1-ee45-49fb-89ca-6081b03060ad 2025-06-24T15:59:50.521923Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2649s 2025-06-24T15:59:50.522798Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:50.610847Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:45Z/6323444041-a991-433d-9a20-08ba787a033c 2025-06-24T15:59:50.611063Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3541s 2025-06-24T15:59:50.611738Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:51.076107Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2211072733.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:59:51.076198Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2211072733.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T15:59:51.076209Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2211072733.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T15:59:51.076256Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:59:51.076244Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T15:59:51.150222Z error klog an error occurred forwarding 42949 -> 15020: error forwarding port 15020 to pod bac434c9af35879f1d8e4b2bba16bf5aa0aee807584c79058bc647b415837175, uid : port forward into network namespace "/var/run/netns/71c2d497-ca45-4dbb-b144-304b49e09198": failed to connect to localhost:15020 inside namespace bac434c9af35879f1d8e4b2bba16bf5aa0aee807584c79058bc647b415837175: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:59:51.152935Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-qhds7 for: Get "http://localhost:42949/debug/ndsz": EOF 2025-06-24T15:59:51.152927Z error port forward failed: lost connection to pod 2025-06-24T15:59:51.153773Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/server-v1-6f45fcb457-qhds7 for: Get "http://localhost:42949/stats/prometheus": dial tcp 127.0.0.1:42949: connect: connection refused 2025-06-24T15:59:51.157509Z error klog an error occurred forwarding 32939 -> 15020: error forwarding port 15020 to pod ce94c91cee8a86b967950aca46edbde08f802bd144033d2ecd56d70da9b19d6c, uid : port forward into network namespace "/var/run/netns/aafc720c-d63c-45e5-8e52-be6fe923e609": failed to connect to localhost:15020 inside namespace ce94c91cee8a86b967950aca46edbde08f802bd144033d2ecd56d70da9b19d6c: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T15:59:51.159206Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-j4msc for: Get "http://localhost:32939/debug/ndsz": EOF 2025-06-24T15:59:51.159244Z error port forward failed: lost connection to pod 2025-06-24T15:59:51.159698Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-j4msc for: Get "http://localhost:32939/stats/prometheus": dial tcp 127.0.0.1:32939: connect: connection refused 2025-06-24T15:59:51.217216Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:45Z/631e33c6d5-3cef-432c-b32c-3bd768b6433e 2025-06-24T15:59:51.217393Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1407s 2025-06-24T15:59:51.217993Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:51.318276Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:45Z/6399697260-76e9-4a96-9be3-d325a4371276 2025-06-24T15:59:51.318559Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2419s 2025-06-24T15:59:51.319158Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:51.441141Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:45Z/63d928033a-0767-44d0-aff6-61ef631a477f 2025-06-24T15:59:51.441307Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3646s 2025-06-24T15:59:51.441916Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:51.729179Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T15:59:51.729253Z info tf === SUCCEEDED: Cleanup Istio in 68.632µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3553250 2025-06-24T16:00:03.219764Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:00:03.219814Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:00:03.253291Z info tf === Test Framework Settings === 2025-06-24T16:00:03.253378Z info tf TestID: telemetry_tracing_otelcollector RunID: 7d887415-3bd1-4998-80dd-ca6edb9d2684 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T16:00:03.253386Z info tf =============================== 2025-06-24T16:00:03.254750Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T16:00:03.255075Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T16:00:03.255088Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T16:00:03.256365Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T16:00:03.256375Z info tf === BEGIN: Building clusters === 2025-06-24T16:00:03.259136Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:00:03.259145Z info tf === DONE: Building clusters === 2025-06-24T16:00:03.259162Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T16:00:03.259514Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:00:03.259523Z info tf === Istio Component Config === 2025-06-24T16:00:03.259560Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T16:00:03.259565Z info tf ================================ 2025-06-24T16:00:03.263903Z info tf skipping deployment as specified in the config 2025-06-24T16:00:03.263921Z info tf === SUCCEEDED: Deploy Istio in 4.403659ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T16:00:03.263927Z info tf === BEGIN: Create namespace echo === 2025-06-24T16:00:03.290582Z info tf === SUCCEEDED: Create namespace echo in 26.636337ms === 2025-06-24T16:00:03.427293Z info tf === BEGIN: Deploy echo instances === 2025-06-24T16:00:03.702426Z info klog Waiting for caches to sync for echo 2025-06-24T16:00:03.726837Z info klog Waiting for caches to sync for echo 2025-06-24T16:00:03.803398Z info klog Caches are synced for echo 2025-06-24T16:00:03.828035Z info klog Caches are synced for echo 2025-06-24T16:00:13.308734Z info tf === SUCCEEDED: Deploy echo instances in 9.881433875s === 2025-06-24T16:00:13.402660Z info tf Checking pods ready... 2025-06-24T16:00:13.402689Z info tf Checking pods ready... 2025-06-24T16:00:13.431363Z info tf [ 0] zipkin-78496844d5-lxpzg Pending (Pending) 2025-06-24T16:00:13.632302Z info tf Checking pods ready... 2025-06-24T16:00:13.632365Z info tf Checking pods ready... 2025-06-24T16:00:13.637929Z info tf [ 0] zipkin-78496844d5-lxpzg Pending (Pending) 2025-06-24T16:00:14.038070Z info tf Checking pods ready... 2025-06-24T16:00:14.038116Z info tf Checking pods ready... 2025-06-24T16:00:14.044383Z info tf [ 0] zipkin-78496844d5-lxpzg Pending (Pending) 2025-06-24T16:00:14.844793Z info tf Checking pods ready... 2025-06-24T16:00:14.844838Z info tf Checking pods ready... 2025-06-24T16:00:14.851342Z info tf [ 0] zipkin-78496844d5-lxpzg Pending (Pending) 2025-06-24T16:00:16.451962Z info tf Checking pods ready... 2025-06-24T16:00:16.451995Z info tf Checking pods ready... 2025-06-24T16:00:16.458670Z info tf [ 0] zipkin-78496844d5-lxpzg Running (container not ready: 'zipkin') 2025-06-24T16:00:19.659183Z info tf Checking pods ready... 2025-06-24T16:00:19.659238Z info tf Checking pods ready... 2025-06-24T16:00:19.664074Z info tf [ 0] zipkin-78496844d5-lxpzg Running (container not ready: 'zipkin') 2025-06-24T16:00:22.864547Z info tf Checking pods ready... 2025-06-24T16:00:22.864587Z info tf Checking pods ready... 2025-06-24T16:00:22.877492Z info tf [ 0] zipkin-78496844d5-lxpzg Running (container not ready: 'zipkin') 2025-06-24T16:00:26.078369Z info tf Checking pods ready... 2025-06-24T16:00:26.078407Z info tf Checking pods ready... 2025-06-24T16:00:26.082472Z info tf [ 0] zipkin-78496844d5-lxpzg Running (Ready) 2025-06-24T16:00:26.239707Z info tf Checking pods ready... 2025-06-24T16:00:26.239750Z info tf Checking pods ready... 2025-06-24T16:00:26.253010Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T16:00:26.453890Z info tf Checking pods ready... 2025-06-24T16:00:26.453935Z info tf Checking pods ready... 2025-06-24T16:00:26.459974Z info tf [ 0] opentelemetry-collector-69596cbbd6-vlbmr Pending (Pending) 2025-06-24T16:00:26.860739Z info tf Checking pods ready... 2025-06-24T16:00:26.860783Z info tf Checking pods ready... 2025-06-24T16:00:26.865331Z info tf [ 0] opentelemetry-collector-69596cbbd6-vlbmr Pending (Pending) 2025-06-24T16:00:27.665407Z info tf Checking pods ready... 2025-06-24T16:00:27.665490Z info tf Checking pods ready... 2025-06-24T16:00:27.670321Z info tf [ 0] opentelemetry-collector-69596cbbd6-vlbmr Pending (Pending) 2025-06-24T16:00:29.271184Z info tf Checking pods ready... 2025-06-24T16:00:29.271217Z info tf Checking pods ready... 2025-06-24T16:00:29.275266Z info tf [ 0] opentelemetry-collector-69596cbbd6-vlbmr Running (Ready) 2025-06-24T16:00:29.325728Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (26.066504496s) === 2025-06-24T16:00:29.327272Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 2025-06-24T16:00:29.338412Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:02:59.339688Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.001274239s)' === 2025-06-24T16:02:59.339781Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T16:02:59.339785Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T16:02:59.579510Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/7186c3b1f2-71ef-4e7d-8907-694e477799bc 2025-06-24T16:02:59.579679Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2385s 2025-06-24T16:02:59.580138Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:02:59.660640Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/7111eea602-440c-4561-bbfd-dbb3775fdb8c 2025-06-24T16:02:59.660876Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3197s 2025-06-24T16:02:59.661410Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:02:59.758043Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/71c54d2370-828c-414f-8acf-c1232ff29b9e 2025-06-24T16:02:59.758132Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4169s 2025-06-24T16:02:59.758694Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (150.95s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (re-run 2) (150.95s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter 2025-06-24T16:00:29.327847Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T16:03:00.290278Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m30.962421679s)' === 2025-06-24T16:03:00.290403Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T16:03:00.290407Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T16:03:00.487139Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/71bec9912e-66ad-4eb1-a36e-02bd59e83d5d 2025-06-24T16:03:00.487476Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1969s 2025-06-24T16:03:00.488014Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:00.611752Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/7141867197-0131-4a33-b2c5-6c618ddd27ae 2025-06-24T16:03:00.611963Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3214s 2025-06-24T16:03:00.612698Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:00.817325Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/71fcc2de12-9562-45b0-b3bb-24652ab5a1a9 2025-06-24T16:03:00.817523Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5268s 2025-06-24T16:03:00.818309Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter (151.84s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter (re-run 2) (151.84s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T16:00:29.327729Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:01:17.633796Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:01:26.112894Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:03:01.169363Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.841628928s)' === 2025-06-24T16:03:01.169478Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:03:01.169475Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:03:01.354049Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/72f7d1506f-6d43-41ce-9eb4-c7e8354ba226 2025-06-24T16:03:01.354476Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1838s 2025-06-24T16:03:01.354939Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:01.546753Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/7205a361d2-6432-48a7-9d37-bbd52fb44640 2025-06-24T16:03:01.547017Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3763s 2025-06-24T16:03:01.547640Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:01.634842Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/72f75b55b2-fa65-4a11-832d-099be06c2218 2025-06-24T16:03:01.635117Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4644s 2025-06-24T16:03:01.635769Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (152.99s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 2) (152.99s) 2025-06-24T16:03:02.315878Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T16:03:02.315936Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m59.056775462s === 2025-06-24T16:03:02.315953Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T16:03:02.316894Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:03:02.316899Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:03:02.524407Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/72216a7b34-e97d-41d4-880c-678af188750f 2025-06-24T16:03:02.524671Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2076s 2025-06-24T16:03:02.525212Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:02.601999Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/724ac837f3-a924-4fe8-b190-e71cc8ac8b3e 2025-06-24T16:03:02.602130Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2850s 2025-06-24T16:03:02.602757Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:02.698726Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/724ad419cf-0ad5-4335-bdcb-a6ea79b0df54 2025-06-24T16:03:02.698876Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3817s 2025-06-24T16:03:02.699567Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:03.533528Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:03:03.533650Z info tf === SUCCEEDED: Cleanup Istio in 105.023µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3554784 2025-06-24T16:03:15.448910Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:03:15.449039Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:03:15.490095Z info tf === Test Framework Settings === 2025-06-24T16:03:15.490193Z info tf TestID: telemetry_tracing_otelcollector RunID: 8146625d-8556-4389-9209-6d84a91ba882 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T16:03:15.490200Z info tf =============================== 2025-06-24T16:03:15.490239Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T16:03:15.490312Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T16:03:15.490326Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T16:03:15.490350Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T16:03:15.490359Z info tf === BEGIN: Building clusters === 2025-06-24T16:03:15.492997Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:03:15.493005Z info tf === DONE: Building clusters === 2025-06-24T16:03:15.493027Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T16:03:15.493057Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:03:15.493066Z info tf === Istio Component Config === 2025-06-24T16:03:15.493119Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T16:03:15.493125Z info tf ================================ 2025-06-24T16:03:15.496543Z info tf skipping deployment as specified in the config 2025-06-24T16:03:15.496559Z info tf === SUCCEEDED: Deploy Istio in 3.498352ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T16:03:15.496567Z info tf === BEGIN: Create namespace echo === 2025-06-24T16:03:15.528146Z info tf === SUCCEEDED: Create namespace echo in 31.567552ms === 2025-06-24T16:03:15.568135Z info tf === BEGIN: Deploy echo instances === 2025-06-24T16:03:15.574644Z error tf === FAILED: Deploy echo instances === 2025-06-24T16:03:15.574657Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.369668620.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:03:15.574679Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:03:15.574691Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:03:15.688520Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:03:15.715555Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:03:15.830806Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-css4f/istio-proxy for: pods "client-cluster-0-v1-5c7cd449cf-css4f" not found 2025-06-24T16:03:15.835528Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-css4f/istio-validation for: pods "client-cluster-0-v1-5c7cd449cf-css4f" not found 2025-06-24T16:03:15.857341Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:09Z/7414392313-28cc-41a6-b37b-b33cedab6191 2025-06-24T16:03:15.857548Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2826s 2025-06-24T16:03:15.858014Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:15.914078Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.369668620.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:03:15.914171Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.369668620.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T16:03:15.914183Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.369668620.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:03:15.914224Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:03:15.914220Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:03:16.022487Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:09Z/74ade64f3b-b3d6-4a05-a441-16466b02809d 2025-06-24T16:03:16.022615Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4476s 2025-06-24T16:03:16.023131Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:16.033633Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:03:16.051812Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:03:16.056735Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:03:16.056813Z info tf === SUCCEEDED: Cleanup Istio in 68.669µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^http_exporter$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3556420 2025-06-24T16:03:27.438187Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:03:27.438245Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:03:27.472640Z info tf === Test Framework Settings === 2025-06-24T16:03:27.472736Z info tf TestID: telemetry_tracing_otelcollector RunID: 861719c0-22b2-458d-9fb0-fb2393a203e5 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T16:03:27.472741Z info tf =============================== 2025-06-24T16:03:27.472772Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T16:03:27.472811Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T16:03:27.472823Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T16:03:27.472841Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T16:03:27.472848Z info tf === BEGIN: Building clusters === 2025-06-24T16:03:27.475559Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:03:27.475570Z info tf === DONE: Building clusters === 2025-06-24T16:03:27.475594Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T16:03:27.475631Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:03:27.475642Z info tf === Istio Component Config === 2025-06-24T16:03:27.475694Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T16:03:27.475703Z info tf ================================ 2025-06-24T16:03:27.479721Z info tf skipping deployment as specified in the config 2025-06-24T16:03:27.479740Z info tf === SUCCEEDED: Deploy Istio in 4.104134ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T16:03:27.479753Z info tf === BEGIN: Create namespace echo === 2025-06-24T16:03:27.503682Z info tf === SUCCEEDED: Create namespace echo in 23.908377ms === 2025-06-24T16:03:27.611429Z info tf === BEGIN: Deploy echo instances === 2025-06-24T16:03:27.968468Z info klog Waiting for caches to sync for echo 2025-06-24T16:03:27.979086Z info klog Waiting for caches to sync for echo 2025-06-24T16:03:28.069199Z info klog Caches are synced for echo 2025-06-24T16:03:28.079863Z info klog Caches are synced for echo 2025-06-24T16:03:37.584738Z info tf === SUCCEEDED: Deploy echo instances in 9.973295028s === 2025-06-24T16:03:37.684629Z info tf Checking pods ready... 2025-06-24T16:03:37.684714Z info tf Checking pods ready... 2025-06-24T16:03:37.712757Z info tf [ 0] zipkin-78496844d5-ssg7l Pending (Pending) 2025-06-24T16:03:37.913582Z info tf Checking pods ready... 2025-06-24T16:03:37.913666Z info tf Checking pods ready... 2025-06-24T16:03:37.917679Z info tf [ 0] zipkin-78496844d5-ssg7l Pending (Pending) 2025-06-24T16:03:38.317916Z info tf Checking pods ready... 2025-06-24T16:03:38.318035Z info tf Checking pods ready... 2025-06-24T16:03:38.323652Z info tf [ 0] zipkin-78496844d5-ssg7l Pending (Pending) 2025-06-24T16:03:39.124452Z info tf Checking pods ready... 2025-06-24T16:03:39.124540Z info tf Checking pods ready... 2025-06-24T16:03:39.128202Z info tf [ 0] zipkin-78496844d5-ssg7l Pending (Pending) 2025-06-24T16:03:40.728335Z info tf Checking pods ready... 2025-06-24T16:03:40.728405Z info tf Checking pods ready... 2025-06-24T16:03:40.732920Z info tf [ 0] zipkin-78496844d5-ssg7l Running (container not ready: 'zipkin') 2025-06-24T16:03:43.933265Z info tf Checking pods ready... 2025-06-24T16:03:43.933372Z info tf Checking pods ready... 2025-06-24T16:03:43.939106Z info tf [ 0] zipkin-78496844d5-ssg7l Running (container not ready: 'zipkin') 2025-06-24T16:03:47.139933Z info tf Checking pods ready... 2025-06-24T16:03:47.140000Z info tf Checking pods ready... 2025-06-24T16:03:47.144838Z info tf [ 0] zipkin-78496844d5-ssg7l Running (container not ready: 'zipkin') 2025-06-24T16:03:50.345129Z info tf Checking pods ready... 2025-06-24T16:03:50.345204Z info tf Checking pods ready... 2025-06-24T16:03:50.349522Z info tf [ 0] zipkin-78496844d5-ssg7l Running (Ready) 2025-06-24T16:03:50.458959Z info tf Checking pods ready... 2025-06-24T16:03:50.459019Z info tf Checking pods ready... 2025-06-24T16:03:50.470753Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T16:03:50.671243Z info tf Checking pods ready... 2025-06-24T16:03:50.671342Z info tf Checking pods ready... 2025-06-24T16:03:50.675400Z info tf [ 0] opentelemetry-collector-69596cbbd6-llqzj Pending (Pending) 2025-06-24T16:03:51.076215Z info tf Checking pods ready... 2025-06-24T16:03:51.076286Z info tf Checking pods ready... 2025-06-24T16:03:51.081096Z info tf [ 0] opentelemetry-collector-69596cbbd6-llqzj Pending (Pending) 2025-06-24T16:03:51.881276Z info tf Checking pods ready... 2025-06-24T16:03:51.881351Z info tf Checking pods ready... 2025-06-24T16:03:51.886226Z info tf [ 0] opentelemetry-collector-69596cbbd6-llqzj Pending (Pending) 2025-06-24T16:03:53.486302Z info tf Checking pods ready... 2025-06-24T16:03:53.486379Z info tf Checking pods ready... 2025-06-24T16:03:53.496083Z info tf [ 0] opentelemetry-collector-69596cbbd6-llqzj Running (Ready) 2025-06-24T16:03:53.538776Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (26.063159637s) === 2025-06-24T16:03:53.538832Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 2025-06-24T16:03:53.553839Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:06:23.554135Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000292086s)' === 2025-06-24T16:06:23.554246Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T16:06:23.554254Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T16:06:23.913314Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/8277f3a292-ea04-432c-9542-4cb6492dbf85 2025-06-24T16:06:23.913632Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3573s 2025-06-24T16:06:23.914305Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.160807Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/8235c822b0-8035-40a0-ad96-12b6379f58d2 2025-06-24T16:06:24.160968Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6046s 2025-06-24T16:06:24.161632Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (150.90s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 2) (150.90s) === RUN TestProxyTracingOpenTelemetryProvider/http_exporter 2025-06-24T16:03:53.542373Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T16:06:24.456501Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.91411879s)' === 2025-06-24T16:06:24.456618Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T16:06:24.456628Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T16:06:24.490932Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/8205a9863f-706d-4603-8948-c30f992851a1 2025-06-24T16:06:24.491275Z info tf istioctl ([x internal-debug --all configz]): completed after 0.9349s 2025-06-24T16:06:24.491895Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.700009Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/82779b349c-50d0-445b-83bc-91263872922f 2025-06-24T16:06:24.700269Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2434s 2025-06-24T16:06:24.700726Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.830243Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/821ac7c176-1bff-43d2-a8a5-299687a04c78 2025-06-24T16:06:24.830481Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3736s 2025-06-24T16:06:24.831190Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.935347Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/824ffbdbc7-ce6a-498d-8440-b57d0bd94a59 2025-06-24T16:06:24.935811Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4789s 2025-06-24T16:06:24.937600Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/http_exporter (151.95s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 2) (151.95s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T16:03:53.542152Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:04:42.902835Z error port forward failed: lost connection to pod 2025-06-24T16:04:43.136411Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward terminated 2025-06-24T16:04:43.169130Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:04:50.377598Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:05:49.172951Z error port forward failed: lost connection to pod 2025-06-24T16:05:49.635658Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward terminated 2025-06-24T16:05:49.672512Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:06:25.492882Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.950723795s)' === 2025-06-24T16:06:25.492988Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:06:25.492987Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:06:26.637464Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/83453a1247-526b-489d-9dc0-b80e4b1b6c95 2025-06-24T16:06:26.637843Z info tf istioctl ([x internal-debug --all configz]): completed after 1.1445s 2025-06-24T16:06:26.638193Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:26.789632Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/83f50deff3-6ef4-483f-a4b5-fa7d09f0aabe 2025-06-24T16:06:26.789926Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.2965s 2025-06-24T16:06:26.790897Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (153.36s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 2) (153.36s) 2025-06-24T16:06:26.907152Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T16:06:26.907258Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m59.431668266s === 2025-06-24T16:06:26.907279Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T16:06:26.907379Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/83249a5365-fcf2-4b34-b126-0dac11c2b50d 2025-06-24T16:06:26.907471Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.4141s 2025-06-24T16:06:26.907564Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:06:26.907561Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:06:26.908044Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:27.352968Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/83bfc1b998-1ca8-4207-807f-de00968de39b 2025-06-24T16:06:27.353136Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4451s 2025-06-24T16:06:27.353878Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:27.569316Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/830f71a21c-1b4b-4425-8fbb-266508c7da5b 2025-06-24T16:06:27.569507Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6615s 2025-06-24T16:06:27.570244Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:27.659189Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/8370184bbc-ccbf-475b-8a3e-decad162865d 2025-06-24T16:06:27.659604Z info tf istioctl ([x internal-debug --all configz]): completed after 0.7515s 2025-06-24T16:06:27.660322Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:28.469592Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:06:28.469787Z info tf === SUCCEEDED: Cleanup Istio in 176.377µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3557970 2025-06-24T16:06:39.643812Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:06:39.643880Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:06:39.680714Z info tf === Test Framework Settings === 2025-06-24T16:06:39.680795Z info tf TestID: telemetry_tracing_otelcollector RunID: ea569070-3d1a-4b62-b964-adadb0a20e13 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T16:06:39.680800Z info tf =============================== 2025-06-24T16:06:39.680831Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T16:06:39.680880Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T16:06:39.680893Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T16:06:39.680916Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T16:06:39.680928Z info tf === BEGIN: Building clusters === 2025-06-24T16:06:39.684182Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:06:39.684191Z info tf === DONE: Building clusters === 2025-06-24T16:06:39.684207Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T16:06:39.684233Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:06:39.684242Z info tf === Istio Component Config === 2025-06-24T16:06:39.684302Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T16:06:39.684311Z info tf ================================ 2025-06-24T16:06:39.691373Z info tf skipping deployment as specified in the config 2025-06-24T16:06:39.691457Z info tf === SUCCEEDED: Deploy Istio in 7.219001ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T16:06:39.691467Z info tf === BEGIN: Create namespace echo === 2025-06-24T16:06:39.706625Z info tf === SUCCEEDED: Create namespace echo in 15.150102ms === 2025-06-24T16:06:39.737110Z info tf === BEGIN: Deploy echo instances === 2025-06-24T16:06:39.744525Z error tf === FAILED: Deploy echo instances === 2025-06-24T16:06:39.744569Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2199101555.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:06:39.744602Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:06:39.744632Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:06:39.841266Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:34Z/85759e801c-de37-40a3-8b0a-e85511c33e4f 2025-06-24T16:06:39.841479Z info tf istioctl ([x internal-debug --all configz]): completed after 0.0964s 2025-06-24T16:06:39.841968Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:39.843506Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:06:39.859386Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:06:39.920495Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:34Z/85a571cf9c-37c4-40f1-8729-f33816fff458 2025-06-24T16:06:39.920770Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1757s 2025-06-24T16:06:39.921482Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:40.012310Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:34Z/85d620e1c0-f19a-464a-8c96-9d816b90254b 2025-06-24T16:06:40.012489Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2674s 2025-06-24T16:06:40.013063Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:40.265238Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-p7r8b/app for: pods "server-v1-6f45fcb457-p7r8b" not found 2025-06-24T16:06:40.270534Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-p7r8b/istio-proxy for: pods "server-v1-6f45fcb457-p7r8b" not found 2025-06-24T16:06:40.273759Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-p7r8b/istio-validation for: pods "server-v1-6f45fcb457-p7r8b" not found 2025-06-24T16:06:40.278172Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2199101555.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:06:40.278185Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2199101555.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T16:06:40.278195Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.2199101555.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:06:40.278231Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:06:40.278281Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:06:40.294860Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-zsdvr/app for: pods "client-cluster-0-v1-5c7cd449cf-zsdvr" not found 2025-06-24T16:06:40.314590Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-zsdvr/istio-proxy for: pods "client-cluster-0-v1-5c7cd449cf-zsdvr" not found 2025-06-24T16:06:40.327567Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/client-cluster-0-v1-5c7cd449cf-zsdvr/istio-validation for: pods "client-cluster-0-v1-5c7cd449cf-zsdvr" not found 2025-06-24T16:06:40.378525Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "client-cluster-0-v1-5c7cd449cf-zsdvr" not found in the "echo" namespace) 2025-06-24T16:06:40.379203Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "client-cluster-0-v1-5c7cd449cf-zsdvr" not found in the "echo" namespace) 2025-06-24T16:06:40.384688Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:06:40.384719Z info tf === SUCCEEDED: Cleanup Istio in 25.425µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^resource_detectors$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3559230 2025-06-24T16:06:52.023904Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:06:52.023972Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:06:52.059243Z info tf === Test Framework Settings === 2025-06-24T16:06:52.059410Z info tf TestID: telemetry_tracing_otelcollector RunID: e5768892-8638-466c-960f-63c41d9069b5 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T16:06:52.059421Z info tf =============================== 2025-06-24T16:06:52.059464Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T16:06:52.059530Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T16:06:52.059548Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T16:06:52.059575Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T16:06:52.059590Z info tf === BEGIN: Building clusters === 2025-06-24T16:06:52.062449Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:06:52.062460Z info tf === DONE: Building clusters === 2025-06-24T16:06:52.062491Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T16:06:52.062524Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:06:52.062534Z info tf === Istio Component Config === 2025-06-24T16:06:52.062591Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T16:06:52.062602Z info tf ================================ 2025-06-24T16:06:52.066579Z info tf skipping deployment as specified in the config 2025-06-24T16:06:52.067480Z info tf === SUCCEEDED: Deploy Istio in 4.907322ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T16:06:52.068084Z info tf === BEGIN: Create namespace echo === 2025-06-24T16:06:52.090654Z info tf === SUCCEEDED: Create namespace echo in 22.557878ms === 2025-06-24T16:06:52.175281Z info tf === BEGIN: Deploy echo instances === 2025-06-24T16:06:52.449665Z info klog Waiting for caches to sync for echo 2025-06-24T16:06:52.451427Z info klog Waiting for caches to sync for echo 2025-06-24T16:06:52.550855Z info klog Caches are synced for echo 2025-06-24T16:06:52.552337Z info klog Caches are synced for echo 2025-06-24T16:07:02.056531Z info tf === SUCCEEDED: Deploy echo instances in 9.881238939s === 2025-06-24T16:07:02.163614Z info tf Checking pods ready... 2025-06-24T16:07:02.163668Z info tf Checking pods ready... 2025-06-24T16:07:02.178329Z info tf [ 0] zipkin-78496844d5-5pmw7 Pending (Pending) 2025-06-24T16:07:02.379231Z info tf Checking pods ready... 2025-06-24T16:07:02.379319Z info tf Checking pods ready... 2025-06-24T16:07:02.384405Z info tf [ 0] zipkin-78496844d5-5pmw7 Pending (Pending) 2025-06-24T16:07:02.784812Z info tf Checking pods ready... 2025-06-24T16:07:02.784910Z info tf Checking pods ready... 2025-06-24T16:07:02.789729Z info tf [ 0] zipkin-78496844d5-5pmw7 Pending (Pending) 2025-06-24T16:07:03.590595Z info tf Checking pods ready... 2025-06-24T16:07:03.590761Z info tf Checking pods ready... 2025-06-24T16:07:03.595387Z info tf [ 0] zipkin-78496844d5-5pmw7 Pending (Pending) 2025-06-24T16:07:05.195439Z info tf Checking pods ready... 2025-06-24T16:07:05.195496Z info tf Checking pods ready... 2025-06-24T16:07:05.199593Z info tf [ 0] zipkin-78496844d5-5pmw7 Running (container not ready: 'zipkin') 2025-06-24T16:07:08.400003Z info tf Checking pods ready... 2025-06-24T16:07:08.400110Z info tf Checking pods ready... 2025-06-24T16:07:08.404835Z info tf [ 0] zipkin-78496844d5-5pmw7 Running (container not ready: 'zipkin') 2025-06-24T16:07:11.605524Z info tf Checking pods ready... 2025-06-24T16:07:11.605596Z info tf Checking pods ready... 2025-06-24T16:07:11.610626Z info tf [ 0] zipkin-78496844d5-5pmw7 Running (container not ready: 'zipkin') 2025-06-24T16:07:14.811573Z info tf Checking pods ready... 2025-06-24T16:07:14.811656Z info tf Checking pods ready... 2025-06-24T16:07:14.817079Z info tf [ 0] zipkin-78496844d5-5pmw7 Running (Ready) 2025-06-24T16:07:14.957126Z info tf Checking pods ready... 2025-06-24T16:07:14.957195Z info tf Checking pods ready... 2025-06-24T16:07:14.968131Z info tf Failed retrieving pods: no matching pod found for selectors: [app=opentelemetry-collector] 2025-06-24T16:07:15.169372Z info tf Checking pods ready... 2025-06-24T16:07:15.169574Z info tf Checking pods ready... 2025-06-24T16:07:15.188725Z info tf [ 0] opentelemetry-collector-69596cbbd6-k2g84 Pending (Pending) 2025-06-24T16:07:15.589972Z info tf Checking pods ready... 2025-06-24T16:07:15.590108Z info tf Checking pods ready... 2025-06-24T16:07:15.595045Z info tf [ 0] opentelemetry-collector-69596cbbd6-k2g84 Pending (Pending) 2025-06-24T16:07:16.395709Z info tf Checking pods ready... 2025-06-24T16:07:16.395795Z info tf Checking pods ready... 2025-06-24T16:07:16.405447Z info tf [ 0] opentelemetry-collector-69596cbbd6-k2g84 Pending (Pending) 2025-06-24T16:07:18.006200Z info tf Checking pods ready... 2025-06-24T16:07:18.006272Z info tf Checking pods ready... 2025-06-24T16:07:18.011122Z info tf [ 0] opentelemetry-collector-69596cbbd6-k2g84 Running (Ready) 2025-06-24T16:07:18.093080Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (26.030567364s) === 2025-06-24T16:07:18.093151Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 2025-06-24T16:07:18.109452Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:09:48.110598Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.001139893s)' === 2025-06-24T16:09:48.110800Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T16:09:48.110796Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T16:09:48.463304Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93f0a54d6f-6fe3-44a6-aa4f-8d719345ff78 2025-06-24T16:09:48.463579Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3519s 2025-06-24T16:09:48.464292Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:48.555006Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93583c1868-a7e0-4a3e-8054-0e352fd39ce5 2025-06-24T16:09:48.555316Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4436s 2025-06-24T16:09:48.556017Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:48.660811Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/9311957f9d-53e9-4f55-b379-66516eb88749 2025-06-24T16:09:48.661168Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5494s 2025-06-24T16:09:48.661754Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (151.06s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (re-run 2) (151.06s) === RUN TestProxyTracingOpenTelemetryProvider/resource_detectors 2025-06-24T16:07:18.093630Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T16:09:49.166881Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m31.073243575s)' === 2025-06-24T16:09:49.167005Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T16:09:49.167002Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T16:09:49.308272Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93c9661aae-24da-4038-91e3-27c9d2cdbec8 2025-06-24T16:09:49.308734Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1415s 2025-06-24T16:09:49.309216Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:49.395543Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93a0167b25-95c2-4df6-9470-32c916626ffb 2025-06-24T16:09:49.395753Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2285s 2025-06-24T16:09:49.396675Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:49.481159Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/930d3f01f6-f57b-4c3a-8b7e-d3d66ef97b41 2025-06-24T16:09:49.481380Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3141s 2025-06-24T16:09:49.481982Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/resource_detectors (151.92s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/resource_detectors (re-run 2) (151.92s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T16:07:18.093468Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:08:07.581465Z error port forward failed: lost connection to pod 2025-06-24T16:08:07.594840Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward terminated 2025-06-24T16:08:07.635194Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:08:14.848660Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:09:13.640388Z error port forward failed: lost connection to pod 2025-06-24T16:09:14.095195Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward terminated 2025-06-24T16:09:14.131497Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:09:50.009984Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.91650986s)' === 2025-06-24T16:09:50.010128Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:09:50.010122Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:09:50.213809Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/941e765fc6-6c66-4423-a4f1-3534256a118c 2025-06-24T16:09:50.214216Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2037s 2025-06-24T16:09:50.214655Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:50.459205Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/94eff98bf8-09ac-4218-b903-6a35bb5c6320 2025-06-24T16:09:50.459494Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4490s 2025-06-24T16:09:50.460067Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:50.581707Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/94e47a3fd2-f986-4dfb-a981-99f655f3e54b 2025-06-24T16:09:50.581958Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5714s 2025-06-24T16:09:50.582708Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (153.17s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 2) (153.17s) 2025-06-24T16:09:51.266565Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T16:09:51.266616Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m59.204136541s === 2025-06-24T16:09:51.266631Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T16:09:51.266844Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:09:51.266843Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:09:51.398157Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/94506d9211-0254-44e5-80e9-a4075faaf529 2025-06-24T16:09:51.398434Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1313s 2025-06-24T16:09:51.399110Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:51.478558Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/94bf1a6ef0-f37a-4370-b35b-156406ede9d6 2025-06-24T16:09:51.478775Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2116s 2025-06-24T16:09:51.479461Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:51.572464Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/945d9190ec-ca1e-4950-ac48-2acd9bbcfa76 2025-06-24T16:09:51.572642Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3055s 2025-06-24T16:09:51.573146Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:52.401902Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:09:52.402025Z info tf === SUCCEEDED: Cleanup Istio in 103.447µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata/primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3560656 2025-06-24T16:10:04.081267Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:10:04.081322Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:10:04.113117Z info tf === Test Framework Settings === 2025-06-24T16:10:04.113217Z info tf TestID: telemetry_tracing_otelcollector RunID: bbb5963a-e679-4e4b-a9d6-be0d521443d3 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T16:10:04.113222Z info tf =============================== 2025-06-24T16:10:04.113461Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T16:10:04.113501Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T16:10:04.113513Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T16:10:04.113530Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T16:10:04.113538Z info tf === BEGIN: Building clusters === 2025-06-24T16:10:04.116281Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:10:04.116293Z info tf === DONE: Building clusters === 2025-06-24T16:10:04.116321Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T16:10:04.116352Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:10:04.116363Z info tf === Istio Component Config === 2025-06-24T16:10:04.116407Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T16:10:04.116414Z info tf ================================ 2025-06-24T16:10:04.119919Z info tf skipping deployment as specified in the config 2025-06-24T16:10:04.119945Z info tf === SUCCEEDED: Deploy Istio in 3.584015ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T16:10:04.119958Z info tf === BEGIN: Create namespace echo === 2025-06-24T16:10:04.134333Z info tf === SUCCEEDED: Create namespace echo in 14.364231ms === 2025-06-24T16:10:04.166223Z info tf === BEGIN: Deploy echo instances === 2025-06-24T16:10:04.179716Z error tf === FAILED: Deploy echo instances === 2025-06-24T16:10:04.179784Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4161982667.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:10:04.179827Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:10:04.179838Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:10:04.290233Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:10:04.291362Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T16:10:04.341594Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:58Z/96336a2efc-726c-4477-8ab4-2a8cfc74b975 2025-06-24T16:10:04.341989Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1617s 2025-06-24T16:10:04.342439Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:10:04.484604Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-zpzlq/app for: pods "server-v1-6f45fcb457-zpzlq" not found 2025-06-24T16:10:04.489458Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-zpzlq/istio-proxy for: pods "server-v1-6f45fcb457-zpzlq" not found 2025-06-24T16:10:04.492603Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/server-v1-6f45fcb457-zpzlq/istio-validation for: pods "server-v1-6f45fcb457-zpzlq" not found 2025-06-24T16:10:04.497421Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4161982667.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:10:04.497486Z info tf === FAILED: Setup: 'telemetry_tracing_otelcollector' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4161982667.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-06-24T16:10:04.497493Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector-/_suite_context/Service.4161982667.yaml] to ns echo in cluster cluster-0: apply: patch: services "client-cluster-0" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-06-24T16:10:04.497532Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:10:04.497526Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:10:04.534010Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:58Z/964dc4ffa4-2a4c-4e4d-b44a-df40af5a2e28 2025-06-24T16:10:04.534303Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3540s 2025-06-24T16:10:04.535162Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:10:04.593812Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "client-cluster-0-v1-5c7cd449cf-zk8dj" not found in the "echo" namespace) 2025-06-24T16:10:04.600044Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "client-cluster-0-v1-5c7cd449cf-zk8dj" not found in the "echo" namespace) 2025-06-24T16:10:04.605741Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:10:04.605767Z info tf === SUCCEEDED: Cleanup Istio in 21.648µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector exec: [go test -json -test.run=^TestProxyTracingOpenTelemetryProvider$/^grpc_exporter_with_initial_metadata$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/tracing/otelcollector -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts] go test pid: 3561970 2025-06-24T16:10:17.194863Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:10:17.194920Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T16:10:17.231417Z info tf === Test Framework Settings === 2025-06-24T16:10:17.231487Z info tf TestID: telemetry_tracing_otelcollector RunID: e926777a-b9b7-4520-87c9-ce29bf7c609f NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T16:10:17.231493Z info tf =============================== 2025-06-24T16:10:17.231522Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/telemetry-tracing-otelcollector- 2025-06-24T16:10:17.231556Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T16:10:17.231565Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T16:10:17.231581Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T16:10:17.231587Z info tf === BEGIN: Building clusters === 2025-06-24T16:10:17.233878Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:10:17.233887Z info tf === DONE: Building clusters === 2025-06-24T16:10:17.233900Z info tf === BEGIN: Setup: 'telemetry_tracing_otelcollector' === 2025-06-24T16:10:17.233919Z info tf === BEGIN: Deploy Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:10:17.233924Z info tf === Istio Component Config === 2025-06-24T16:10:17.233950Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T16:10:17.233954Z info tf ================================ 2025-06-24T16:10:17.239239Z info tf skipping deployment as specified in the config 2025-06-24T16:10:17.239335Z info tf === SUCCEEDED: Deploy Istio in 5.407494ms [Suite=telemetry_tracing_otelcollector]=== 2025-06-24T16:10:17.239347Z info tf === BEGIN: Create namespace echo === 2025-06-24T16:10:17.258686Z info tf === SUCCEEDED: Create namespace echo in 19.335802ms === 2025-06-24T16:10:17.341840Z info tf === BEGIN: Deploy echo instances === 2025-06-24T16:10:17.611010Z info klog Waiting for caches to sync for echo 2025-06-24T16:10:17.611091Z info klog Waiting for caches to sync for echo 2025-06-24T16:10:17.712196Z info klog Caches are synced for echo 2025-06-24T16:10:17.712336Z info klog Caches are synced for echo 2025-06-24T16:10:27.217649Z info tf === SUCCEEDED: Deploy echo instances in 9.875800232s === 2025-06-24T16:10:27.367681Z info tf Checking pods ready... 2025-06-24T16:10:27.367731Z info tf Checking pods ready... 2025-06-24T16:10:27.397974Z info tf [ 0] zipkin-78496844d5-t9tpx Pending (Pending) 2025-06-24T16:10:27.598792Z info tf Checking pods ready... 2025-06-24T16:10:27.598874Z info tf Checking pods ready... 2025-06-24T16:10:27.604805Z info tf [ 0] zipkin-78496844d5-t9tpx Pending (Pending) 2025-06-24T16:10:28.005224Z info tf Checking pods ready... 2025-06-24T16:10:28.005311Z info tf Checking pods ready... 2025-06-24T16:10:28.009664Z info tf [ 0] zipkin-78496844d5-t9tpx Pending (Pending) 2025-06-24T16:10:28.810167Z info tf Checking pods ready... 2025-06-24T16:10:28.810245Z info tf Checking pods ready... 2025-06-24T16:10:28.814572Z info tf [ 0] zipkin-78496844d5-t9tpx Pending (Pending) 2025-06-24T16:10:30.415150Z info tf Checking pods ready... 2025-06-24T16:10:30.415184Z info tf Checking pods ready... 2025-06-24T16:10:30.419158Z info tf [ 0] zipkin-78496844d5-t9tpx Running (container not ready: 'zipkin') 2025-06-24T16:10:33.619267Z info tf Checking pods ready... 2025-06-24T16:10:33.619297Z info tf Checking pods ready... 2025-06-24T16:10:33.623938Z info tf [ 0] zipkin-78496844d5-t9tpx Running (container not ready: 'zipkin') 2025-06-24T16:10:36.824057Z info tf Checking pods ready... 2025-06-24T16:10:36.824090Z info tf Checking pods ready... 2025-06-24T16:10:36.830987Z info tf [ 0] zipkin-78496844d5-t9tpx Running (container not ready: 'zipkin') 2025-06-24T16:10:40.032078Z info tf Checking pods ready... 2025-06-24T16:10:40.032129Z info tf Checking pods ready... 2025-06-24T16:10:40.044157Z info tf [ 0] zipkin-78496844d5-t9tpx Running (Ready) 2025-06-24T16:10:40.559556Z info tf Checking pods ready... 2025-06-24T16:10:40.559586Z info tf Checking pods ready... 2025-06-24T16:10:40.607706Z info tf [ 0] opentelemetry-collector-69596cbbd6-t4zzb Pending (Pending) 2025-06-24T16:10:40.808562Z info tf Checking pods ready... 2025-06-24T16:10:40.808642Z info tf Checking pods ready... 2025-06-24T16:10:40.816560Z info tf [ 0] opentelemetry-collector-69596cbbd6-t4zzb Pending (Pending) 2025-06-24T16:10:41.216969Z info tf Checking pods ready... 2025-06-24T16:10:41.217038Z info tf Checking pods ready... 2025-06-24T16:10:41.226074Z info tf [ 0] opentelemetry-collector-69596cbbd6-t4zzb Pending (Pending) 2025-06-24T16:10:42.026164Z info tf Checking pods ready... 2025-06-24T16:10:42.026220Z info tf Checking pods ready... 2025-06-24T16:10:42.030554Z info tf [ 0] opentelemetry-collector-69596cbbd6-t4zzb Pending (Pending) 2025-06-24T16:10:43.631139Z info tf Checking pods ready... 2025-06-24T16:10:43.631185Z info tf Checking pods ready... 2025-06-24T16:10:43.635077Z info tf [ 0] opentelemetry-collector-69596cbbd6-t4zzb Running (Ready) 2025-06-24T16:10:43.680152Z info tf === DONE: Setup: 'telemetry_tracing_otelcollector' (26.446230057s) === 2025-06-24T16:10:43.680240Z info tf === BEGIN: Test Run: 'telemetry_tracing_otelcollector' === === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 2025-06-24T16:10:43.690825Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:13:13.691262Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.00043441s)' === 2025-06-24T16:13:13.691376Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T16:13:13.691428Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T16:13:13.804721Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/104ee996096-1da0-4254-9ec4-a195f94c05a2 2025-06-24T16:13:13.804904Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1129s 2025-06-24T16:13:13.805543Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:13.949724Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/1048e0d3ab1-e77d-4842-988d-484291935fe9 2025-06-24T16:13:13.950021Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2581s 2025-06-24T16:13:13.950512Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:14.012258Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/10461a66eee-e62b-403c-8da3-2e2afe89635f 2025-06-24T16:13:14.012356Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3204s 2025-06-24T16:13:14.012815Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (150.75s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 2) (150.75s) === RUN TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata 2025-06-24T16:10:43.680904Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T16:13:14.436263Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.755354543s)' === 2025-06-24T16:13:14.436338Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T16:13:14.436341Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T16:13:14.561349Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/104f2709652-4389-4e80-bc61-06c91a0e74c9 2025-06-24T16:13:14.561454Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1249s 2025-06-24T16:13:14.562018Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:14.635140Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/10402aca392-aa42-4091-882f-917636eceac9 2025-06-24T16:13:14.635468Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1989s 2025-06-24T16:13:14.635867Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:14.740561Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/10428173ec8-00c0-4228-b998-b1aa5187a67d 2025-06-24T16:13:14.740627Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3040s 2025-06-24T16:13:14.741307Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (151.56s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 2) (151.56s) === RUN TestProxyTracingOpenTelemetryProvider 2025-06-24T16:10:43.680537Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:11:31.621749Z error port forward failed: lost connection to pod 2025-06-24T16:11:31.864353Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward terminated 2025-06-24T16:11:31.904228Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:11:40.126541Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:12:37.912958Z error port forward failed: lost connection to pod 2025-06-24T16:12:38.364644Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward terminated 2025-06-24T16:12:38.414360Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:13:15.236794Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.556254496s)' === 2025-06-24T16:13:15.236866Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:13:15.236865Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:13:15.363567Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/105b56be1ae-1e06-4d7b-9c86-ae9f4588f212 2025-06-24T16:13:15.363805Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1267s 2025-06-24T16:13:15.364405Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:15.465671Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/1059b08b3eb-a570-4133-bae2-da9fea991867 2025-06-24T16:13:15.465815Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2287s 2025-06-24T16:13:15.466442Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:15.565291Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/10541f7d8ee-034b-4648-bbb7-bb52cffdc1bc 2025-06-24T16:13:15.565443Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3283s 2025-06-24T16:13:15.565856Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestProxyTracingOpenTelemetryProvider (152.86s) FAIL tests/integration/telemetry/tracing/otelcollector.TestProxyTracingOpenTelemetryProvider (re-run 2) (152.86s) 2025-06-24T16:13:16.543209Z info tf === FAILED: Test Run: 'telemetry_tracing_otelcollector' (exitCode: 1) === 2025-06-24T16:13:16.543256Z info tf === Suite "telemetry_tracing_otelcollector" run time: 2m59.309358923s === 2025-06-24T16:13:16.543267Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-tracing-otelcollector/artifacts/trace.yaml 2025-06-24T16:13:16.543428Z error tf === Dumping Namespace echo State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:13:16.543443Z error tf === Dumping Istio Deployment State for [suite(telemetry_tracing_otelcollector)]... 2025-06-24T16:13:16.859065Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/10587fc26a6-7db1-4d85-b2a2-d4b5def28213 2025-06-24T16:13:16.859319Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3157s 2025-06-24T16:13:16.859831Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:16.946116Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/1051a3aef7f-939e-4226-94b1-e11b81033386 2025-06-24T16:13:16.946290Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4027s 2025-06-24T16:13:16.946967Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:17.041208Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/105a103c573-1e40-4a8f-9b7b-19015abaa6b9 2025-06-24T16:13:17.041315Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4977s 2025-06-24T16:13:17.041918Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:17.778792Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_tracing_otelcollector] === 2025-06-24T16:13:17.778910Z info tf === SUCCEEDED: Cleanup Istio in 102.941µs [Suite=telemetry_tracing_otelcollector] === FAIL tests/integration/telemetry/tracing/otelcollector === Failed === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (150.69s) 2025-06-24T15:35:53.781163Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:38:23.781622Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.000455019s)' === 2025-06-24T15:38:23.781713Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:38:23.781709Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:38:23.947198Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/12d286231e-945b-41df-98b8-387a7090060c 2025-06-24T15:38:23.947717Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1653s 2025-06-24T15:38:23.948095Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.039991Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/1290404357-75a6-41d0-809c-5afe42f52286 2025-06-24T15:38:24.040139Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2577s 2025-06-24T15:38:24.040632Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.124277Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/12523ed408-d217-4650-a9f8-45952ebfe7da 2025-06-24T15:38:24.124385Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3419s 2025-06-24T15:38:24.125031Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter (151.56s) 2025-06-24T15:35:53.771730Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T15:38:24.475702Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m30.703969388s)' === 2025-06-24T15:38:24.475775Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:38:24.475788Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:38:24.611695Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/129b79c1ff-69b3-4159-8e5a-458bb8c1e900 2025-06-24T15:38:24.612074Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1361s 2025-06-24T15:38:24.612437Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.706678Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/121cc6233f-c377-445b-9f55-6547b261d253 2025-06-24T15:38:24.706812Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2309s 2025-06-24T15:38:24.707413Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:38:24.811488Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:35:53Z/1207fd7836-2a3d-4255-8046-0fcb414769a1 2025-06-24T15:38:24.811662Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3356s 2025-06-24T15:38:24.812254Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (151.14s) 2025-06-24T15:38:25.344967Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:40:55.345947Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000976684s)' === 2025-06-24T15:40:55.346052Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:40:55.346315Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:40:56.317496Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/13abc8ecd7-59cf-4b56-803b-611388f0b9d1 2025-06-24T15:40:56.317788Z info tf istioctl ([x internal-debug --all configz]): completed after 0.9711s 2025-06-24T15:40:56.318274Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter (151.96s) 2025-06-24T15:38:25.331127Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T15:40:56.479480Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m31.148347397s)' === 2025-06-24T15:40:56.479573Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:40:56.479572Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:40:56.488755Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/13ded7bbed-77ef-43fd-a52c-4279d23df06f 2025-06-24T15:40:56.489508Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1427s 2025-06-24T15:40:56.492385Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:40:56.710901Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/133c764846-fb32-410c-97a7-80be85e3857b 2025-06-24T15:40:56.711272Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.3644s 2025-06-24T15:40:56.711741Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:40:57.127642Z error port forward failed: lost connection to pod 2025-06-24T15:40:57.164622Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/13dc807926-dc04-4f67-bea9-fd334247277d 2025-06-24T15:40:57.164906Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6851s 2025-06-24T15:40:57.165437Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (150.94s) 2025-06-24T15:40:57.300962Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:43:27.301537Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.000575062s)' === 2025-06-24T15:43:27.301654Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:43:27.301649Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:43:27.538617Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/144b0318a1-a3fa-4c6d-b578-026eee176981 2025-06-24T15:43:27.538734Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2364s 2025-06-24T15:43:27.539555Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:27.608720Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/14341c8c98-525c-4ee5-9bc9-8089c21785b4 2025-06-24T15:43:27.608946Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3066s 2025-06-24T15:43:27.609566Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:27.695701Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/14f655a400-756d-4436-90f7-1eac0d7296df 2025-06-24T15:43:27.695849Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3934s 2025-06-24T15:43:27.696622Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors (152.05s) 2025-06-24T15:40:57.288128Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T15:43:28.238319Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m30.950181002s)' === 2025-06-24T15:43:28.238436Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:43:28.238430Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:43:28.446980Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/14609cb876-3107-4116-b271-cac586fbe721 2025-06-24T15:43:28.447330Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2086s 2025-06-24T15:43:28.447783Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:28.570459Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/1499dde033-77cb-471d-81a4-91ce3f3744ac 2025-06-24T15:43:28.570677Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3319s 2025-06-24T15:43:28.571487Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:43:28.687799Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/142f7902c1-46b1-4fb3-b628-1e130a667793 2025-06-24T15:43:28.687972Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4492s 2025-06-24T15:43:28.688487Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (151.26s) 2025-06-24T15:43:29.354322Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:45:59.355303Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.000980727s)' === 2025-06-24T15:45:59.355415Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:45:59.355424Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:45:59.553603Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15e874dedd-eda1-42fc-82cc-e7d3ef24f92d 2025-06-24T15:45:59.553970Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1974s 2025-06-24T15:45:59.554847Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:45:59.714043Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15b52a17fe-f863-439d-acec-cf9385e9094a 2025-06-24T15:45:59.714158Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3576s 2025-06-24T15:45:59.714896Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:45:59.797060Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/159fb613d2-b2ee-4ead-893a-fcbea8659252 2025-06-24T15:45:59.797190Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4406s 2025-06-24T15:45:59.797785Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (152.43s) 2025-06-24T15:43:29.337087Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T15:46:00.617192Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m31.280099531s)' === 2025-06-24T15:46:00.617305Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:46:00.617304Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:46:00.746891Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15734335d3-02c4-41db-987f-994818f61adf 2025-06-24T15:46:00.747179Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1297s 2025-06-24T15:46:00.747590Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:00.879582Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/1585f7ba94-1e5f-418c-9056-922044316395 2025-06-24T15:46:00.879779Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2623s 2025-06-24T15:46:00.880354Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:00.956424Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:43:29Z/15e8dfb61a-e8fe-462c-9d62-73ef46ef13c9 2025-06-24T15:46:00.956576Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3390s 2025-06-24T15:46:00.957061Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (609.37s) 2025-06-24T15:35:53.771571Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:36:31.430751Z error port forward failed: lost connection to pod 2025-06-24T15:36:31.585073Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:36:31.631149Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:36:44.153686Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:37:37.635973Z error port forward failed: lost connection to pod 2025-06-24T15:37:38.085734Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:37:38.126855Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:38:44.132160Z error port forward failed: lost connection to pod 2025-06-24T15:38:44.586033Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:38:44.617740Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:39:50.622812Z error port forward failed: lost connection to pod 2025-06-24T15:39:51.085652Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward terminated 2025-06-24T15:39:51.122708Z warn pod: echo/server-v1-6f45fcb457-ghkg5 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:40:57.322422Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:38:25Z/133c07ff59-6635-4d97-8b22-66e4ae37a32b 2025-06-24T15:40:57.322510Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8427s 2025-06-24T15:40:57.323137Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:40:57.444697Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:40:57Z/1487b53829-7799-4219-9cd1-a804ae258e7b 2025-06-24T15:40:57.444783Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9649s 2025-06-24T15:40:57.445346Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:46:01.765846Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (10m7.994269038s)' === 2025-06-24T15:46:01.765991Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:46:01.765993Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:46:01.925699Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/161d5ecbec-2c12-4d40-83c0-f2fa81adc02b 2025-06-24T15:46:01.925945Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1591s 2025-06-24T15:46:01.926700Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:02.107230Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/160e745e9b-566b-4a2c-9abf-7b2345977ee2 2025-06-24T15:46:02.107425Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3405s 2025-06-24T15:46:02.108065Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:46:02.202311Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:01Z/16e09e6789-4aae-490e-8c20-e4cdf175f888 2025-06-24T15:46:02.202569Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4356s 2025-06-24T15:46:02.203199Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (re-run 1) (151.79s) 2025-06-24T15:46:55.444042Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:49:25.444772Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.00072479s)' === 2025-06-24T15:49:25.444860Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:49:25.444845Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T15:49:26.668821Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26e581777b-0e23-4f3b-a982-e8e03b3b6ca1 2025-06-24T15:49:26.669187Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.2240s 2025-06-24T15:49:26.669666Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:26.964264Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26e85e0e3d-7c2b-44ea-84cd-2189aa095511 2025-06-24T15:49:26.964750Z info tf istioctl ([x internal-debug --all configz]): completed after 1.5195s 2025-06-24T15:49:26.965172Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:27.078869Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26f9dbe1fd-e96d-45ed-b2c6-7aa7951ca3f7 2025-06-24T15:49:27.079127Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.6338s 2025-06-24T15:49:27.079819Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter (re-run 1) (152.71s) 2025-06-24T15:46:55.434906Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T15:49:27.234869Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m31.799954668s)' === 2025-06-24T15:49:27.235004Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:49:27.235001Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T15:49:27.398161Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26705f23f6-4e44-47af-bdb8-f297ae4f93e3 2025-06-24T15:49:27.398524Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1633s 2025-06-24T15:49:27.399130Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:27.604917Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26de6a1fff-dac3-4d28-b9c8-108764df4724 2025-06-24T15:49:27.605125Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3699s 2025-06-24T15:49:27.605900Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:27.708780Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:46:55Z/26d1244b82-7d75-45a7-8a64-38cda3dc066e 2025-06-24T15:49:27.708953Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4737s 2025-06-24T15:49:27.709655Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 1) (153.86s) 2025-06-24T15:46:55.434786Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:47:45.463224Z error port forward failed: lost connection to pod 2025-06-24T15:47:45.843316Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward terminated 2025-06-24T15:47:45.881526Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:47:52.230403Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:48:51.887443Z error port forward failed: lost connection to pod 2025-06-24T15:48:52.342394Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward terminated 2025-06-24T15:48:52.392561Z warn pod: echo/server-v1-6f45fcb457-zfx47 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:49:28.148200Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m32.713406772s)' === 2025-06-24T15:49:28.148336Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:49:28.148325Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:49:28.463346Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/2761e5df7a-d553-447c-a718-a86f01446545 2025-06-24T15:49:28.463630Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3150s 2025-06-24T15:49:28.464175Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:28.545419Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/27cfde0e8e-f146-4f56-8ea3-ff9644c60344 2025-06-24T15:49:28.545726Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3971s 2025-06-24T15:49:28.546399Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:49:28.628963Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:49:28Z/27595e3772-86d6-40d9-97bd-c77fc7c47e09 2025-06-24T15:49:28.629124Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4805s 2025-06-24T15:49:28.629805Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 1) (150.75s) 2025-06-24T15:50:19.478869Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:52:49.479741Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000863306s)' === 2025-06-24T15:52:49.479892Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:52:49.479864Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T15:52:49.620226Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/3742dceb6f-b1a4-4fb8-b239-5bed0d27183d 2025-06-24T15:52:49.620504Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1399s 2025-06-24T15:52:49.620955Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:49.722291Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/3702bd74c5-ff05-4abf-908d-508405a261c3 2025-06-24T15:52:49.722622Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2420s 2025-06-24T15:52:49.722968Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:49.848277Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/377532dcc3-3294-4ecc-a859-f972512763ba 2025-06-24T15:52:49.848471Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3679s 2025-06-24T15:52:49.848967Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 1) (151.51s) 2025-06-24T15:50:19.467860Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T15:52:50.226797Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.758927532s)' === 2025-06-24T15:52:50.226909Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:52:50.226940Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T15:52:50.338306Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/377ba1f95f-7f80-4070-9d38-ae148dacae71 2025-06-24T15:52:50.338453Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1112s 2025-06-24T15:52:50.339008Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:50.423146Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/3707b1a05f-89eb-4d35-9360-b3e0b2c99ed6 2025-06-24T15:52:50.423374Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.1961s 2025-06-24T15:52:50.423869Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:50.517582Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:50:19Z/37db79d218-1b82-4693-9a23-2a985c147903 2025-06-24T15:52:50.518635Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2913s 2025-06-24T15:52:50.519427Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 1) (152.75s) 2025-06-24T15:50:19.467732Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:51:08.200205Z error port forward failed: lost connection to pod 2025-06-24T15:51:08.545421Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward terminated 2025-06-24T15:51:08.592668Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:51:16.270210Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:52:14.598344Z error port forward failed: lost connection to pod 2025-06-24T15:52:15.045278Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward terminated 2025-06-24T15:52:15.081011Z warn pod: echo/server-v1-6f45fcb457-jjvzz port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:52:50.973643Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.505900602s)' === 2025-06-24T15:52:50.973795Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:52:50.973784Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:52:51.127432Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/38d732b9c5-60a2-474c-afc2-aec8b68cc50c 2025-06-24T15:52:51.127582Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1533s 2025-06-24T15:52:51.128214Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:51.221994Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/389f96d9aa-ddfe-40cb-9ac8-3c19462ec779 2025-06-24T15:52:51.222287Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2480s 2025-06-24T15:52:51.222804Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:52:51.325782Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:52:51Z/38d46cc54b-9011-4101-8ed2-49c1f256f2ba 2025-06-24T15:52:51.325908Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3516s 2025-06-24T15:52:51.326484Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (re-run 1) (150.85s) 2025-06-24T15:53:42.554520Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:56:12.555845Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.001317097s)' === 2025-06-24T15:56:12.555961Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:56:12.555983Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T15:56:12.822215Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/496a49b029-3545-4ef5-9582-a21be7b4612e 2025-06-24T15:56:12.822543Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2662s 2025-06-24T15:56:12.822951Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:12.918124Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/4924a48e35-0b90-42ad-b18a-638038e64542 2025-06-24T15:56:12.918319Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3620s 2025-06-24T15:56:12.918853Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:13.003841Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/49095eeb08-c512-4095-b3f1-6f8ab2738b66 2025-06-24T15:56:13.003974Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4476s 2025-06-24T15:56:13.004534Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors (re-run 1) (151.78s) 2025-06-24T15:53:42.545124Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T15:56:13.400543Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m30.855409295s)' === 2025-06-24T15:56:13.400688Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:56:13.400708Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T15:56:13.786272Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/495c793321-a1a3-411e-a1fd-e4f821bdfec1 2025-06-24T15:56:13.786597Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3856s 2025-06-24T15:56:13.787214Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:13.861288Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/49d1e96c4f-ba30-47db-94d7-d7ab6bcf6238 2025-06-24T15:56:13.861501Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4605s 2025-06-24T15:56:13.862388Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:13.941838Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:53:42Z/493536e542-2b26-4f9c-a589-80ad03678431 2025-06-24T15:56:13.942063Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5411s 2025-06-24T15:56:13.942623Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 1) (153.00s) 2025-06-24T15:53:42.544990Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:54:31.274449Z error port forward failed: lost connection to pod 2025-06-24T15:54:31.544590Z warn pod: echo/server-v1-6f45fcb457-wmmqg port forward terminated 2025-06-24T15:54:31.587086Z warn pod: echo/server-v1-6f45fcb457-wmmqg port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:54:39.359628Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:55:37.592106Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:56:14.327572Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.782577971s)' === 2025-06-24T15:56:14.327660Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:56:14.327659Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:56:14.487637Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/50a739f71e-a38a-4c82-840e-308d46a0bd91 2025-06-24T15:56:14.487941Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1600s 2025-06-24T15:56:14.488476Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:14.590219Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/506fef122f-50fd-4c6c-8fd9-1b75fc0d572b 2025-06-24T15:56:14.590395Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2625s 2025-06-24T15:56:14.591239Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:56:14.672211Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:56:14Z/50f4050e77-a30b-448c-a152-1bc23b6d3cea 2025-06-24T15:56:14.672422Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3445s 2025-06-24T15:56:14.673008Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 1) (150.89s) 2025-06-24T15:57:05.258271Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T15:59:35.259171Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.00089766s)' === 2025-06-24T15:59:35.259323Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:59:35.259317Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T15:59:35.447975Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/6084ba06e5-7c8f-45cb-a6e2-344212da33e5 2025-06-24T15:59:35.448144Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1885s 2025-06-24T15:59:35.448814Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:35.619921Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/60bc80d092-fc12-449e-a83f-5ce328d26793 2025-06-24T15:59:35.620168Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3605s 2025-06-24T15:59:35.620742Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:35.769538Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/60cc569d9b-057e-4b81-b004-0724f8001987 2025-06-24T15:59:35.769753Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5101s 2025-06-24T15:59:35.770504Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 1) (151.77s) 2025-06-24T15:57:05.246853Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T15:59:36.143493Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.896628363s)' === 2025-06-24T15:59:36.143639Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:59:36.143641Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T15:59:36.336299Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/608b36e432-34fd-4eb8-9682-5b1b3b6a3e6e 2025-06-24T15:59:36.336602Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1927s 2025-06-24T15:59:36.337073Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:36.462518Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/602add8c88-57fe-496f-b984-abb99d8186ad 2025-06-24T15:59:36.462856Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3189s 2025-06-24T15:59:36.463659Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:36.599194Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:57:05Z/6067d93c6c-85e8-49a3-a057-e5f6e7a32331 2025-06-24T15:59:36.599552Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4556s 2025-06-24T15:59:36.600351Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 1) (153.09s) 2025-06-24T15:57:05.246574Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:57:54.492304Z error port forward failed: lost connection to pod 2025-06-24T15:57:54.785132Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward terminated 2025-06-24T15:57:54.847919Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:58:02.067787Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:59:00.855656Z error port forward failed: lost connection to pod 2025-06-24T15:59:01.284349Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward terminated 2025-06-24T15:59:01.319909Z warn pod: echo/server-v1-6f45fcb457-qhds7 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T15:59:37.015538Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.768961763s)' === 2025-06-24T15:59:37.015630Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:59:37.015637Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T15:59:37.185382Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/6138f6130e-c775-4271-baa8-28cf994d44f7 2025-06-24T15:59:37.185629Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1697s 2025-06-24T15:59:37.186287Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:37.303504Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/61dfb526ee-1fe1-46be-8abe-70cad589a044 2025-06-24T15:59:37.303743Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2877s 2025-06-24T15:59:37.304301Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T15:59:37.402909Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T15:59:37Z/618331940c-3889-49c2-82ed-8329e07d1bce 2025-06-24T15:59:37.403044Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3870s 2025-06-24T15:59:37.403607Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0 (re-run 2) (150.95s) 2025-06-24T16:00:29.338412Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:02:59.339688Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0] (2m30.001274239s)' === 2025-06-24T16:02:59.339781Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T16:02:59.339785Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter/primary-0... 2025-06-24T16:02:59.579510Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/7186c3b1f2-71ef-4e7d-8907-694e477799bc 2025-06-24T16:02:59.579679Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2385s 2025-06-24T16:02:59.580138Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:02:59.660640Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/7111eea602-440c-4561-bbfd-dbb3775fdb8c 2025-06-24T16:02:59.660876Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3197s 2025-06-24T16:02:59.661410Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:02:59.758043Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/71c54d2370-828c-414f-8acf-c1232ff29b9e 2025-06-24T16:02:59.758132Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4169s 2025-06-24T16:02:59.758694Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter (re-run 2) (151.84s) 2025-06-24T16:00:29.327847Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter]' === 2025-06-24T16:03:00.290278Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter] (2m30.962421679s)' === 2025-06-24T16:03:00.290403Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T16:03:00.290407Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter... 2025-06-24T16:03:00.487139Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/71bec9912e-66ad-4eb1-a36e-02bd59e83d5d 2025-06-24T16:03:00.487476Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1969s 2025-06-24T16:03:00.488014Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:00.611752Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/7141867197-0131-4a33-b2c5-6c618ddd27ae 2025-06-24T16:03:00.611963Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3214s 2025-06-24T16:03:00.612698Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:00.817325Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:00:29Z/71fcc2de12-9562-45b0-b3bb-24652ab5a1a9 2025-06-24T16:03:00.817523Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5268s 2025-06-24T16:03:00.818309Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 2) (152.99s) 2025-06-24T16:00:29.327729Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:01:17.633796Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:01:26.112894Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:03:01.169363Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.841628928s)' === 2025-06-24T16:03:01.169478Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:03:01.169475Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:03:01.354049Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/72f7d1506f-6d43-41ce-9eb4-c7e8354ba226 2025-06-24T16:03:01.354476Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1838s 2025-06-24T16:03:01.354939Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:01.546753Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/7205a361d2-6432-48a7-9d37-bbd52fb44640 2025-06-24T16:03:01.547017Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3763s 2025-06-24T16:03:01.547640Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:03:01.634842Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:01Z/72f75b55b2-fa65-4a11-832d-099be06c2218 2025-06-24T16:03:01.635117Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4644s 2025-06-24T16:03:01.635769Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0 (re-run 2) (150.90s) 2025-06-24T16:03:53.553839Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:06:23.554135Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0] (2m30.000292086s)' === 2025-06-24T16:06:23.554246Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T16:06:23.554254Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter/primary-0... 2025-06-24T16:06:23.913314Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/8277f3a292-ea04-432c-9542-4cb6492dbf85 2025-06-24T16:06:23.913632Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3573s 2025-06-24T16:06:23.914305Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.160807Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/8235c822b0-8035-40a0-ad96-12b6379f58d2 2025-06-24T16:06:24.160968Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6046s 2025-06-24T16:06:24.161632Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/http_exporter (re-run 2) (151.95s) 2025-06-24T16:03:53.542373Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter]' === 2025-06-24T16:06:24.456501Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/http_exporter] (2m30.91411879s)' === 2025-06-24T16:06:24.456618Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T16:06:24.456628Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/http_exporter... 2025-06-24T16:06:24.490932Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/8205a9863f-706d-4603-8948-c30f992851a1 2025-06-24T16:06:24.491275Z info tf istioctl ([x internal-debug --all configz]): completed after 0.9349s 2025-06-24T16:06:24.491895Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.700009Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/82779b349c-50d0-445b-83bc-91263872922f 2025-06-24T16:06:24.700269Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2434s 2025-06-24T16:06:24.700726Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.830243Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/821ac7c176-1bff-43d2-a8a5-299687a04c78 2025-06-24T16:06:24.830481Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3736s 2025-06-24T16:06:24.831190Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:24.935347Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:03:53Z/824ffbdbc7-ce6a-498d-8440-b57d0bd94a59 2025-06-24T16:06:24.935811Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4789s 2025-06-24T16:06:24.937600Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 2) (153.36s) 2025-06-24T16:03:53.542152Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:04:42.902835Z error port forward failed: lost connection to pod 2025-06-24T16:04:43.136411Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward terminated 2025-06-24T16:04:43.169130Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:04:50.377598Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:05:49.172951Z error port forward failed: lost connection to pod 2025-06-24T16:05:49.635658Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward terminated 2025-06-24T16:05:49.672512Z warn pod: echo/server-v1-6f45fcb457-p7r8b port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:06:25.492882Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.950723795s)' === 2025-06-24T16:06:25.492988Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:06:25.492987Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:06:26.637464Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/83453a1247-526b-489d-9dc0-b80e4b1b6c95 2025-06-24T16:06:26.637843Z info tf istioctl ([x internal-debug --all configz]): completed after 1.1445s 2025-06-24T16:06:26.638193Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:06:26.789632Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:06:25Z/83f50deff3-6ef4-483f-a4b5-fa7d09f0aabe 2025-06-24T16:06:26.789926Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.2965s 2025-06-24T16:06:26.790897Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0 (re-run 2) (151.06s) 2025-06-24T16:07:18.109452Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:09:48.110598Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0] (2m30.001139893s)' === 2025-06-24T16:09:48.110800Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T16:09:48.110796Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors/primary-0... 2025-06-24T16:09:48.463304Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93f0a54d6f-6fe3-44a6-aa4f-8d719345ff78 2025-06-24T16:09:48.463579Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3519s 2025-06-24T16:09:48.464292Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:48.555006Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93583c1868-a7e0-4a3e-8054-0e352fd39ce5 2025-06-24T16:09:48.555316Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4436s 2025-06-24T16:09:48.556017Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:48.660811Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/9311957f9d-53e9-4f55-b379-66516eb88749 2025-06-24T16:09:48.661168Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5494s 2025-06-24T16:09:48.661754Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/resource_detectors (re-run 2) (151.92s) 2025-06-24T16:07:18.093630Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors]' === 2025-06-24T16:09:49.166881Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/resource_detectors] (2m31.073243575s)' === 2025-06-24T16:09:49.167005Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T16:09:49.167002Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/resource_detectors... 2025-06-24T16:09:49.308272Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93c9661aae-24da-4038-91e3-27c9d2cdbec8 2025-06-24T16:09:49.308734Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1415s 2025-06-24T16:09:49.309216Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:49.395543Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/93a0167b25-95c2-4df6-9470-32c916626ffb 2025-06-24T16:09:49.395753Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2285s 2025-06-24T16:09:49.396675Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:49.481159Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:07:18Z/930d3f01f6-f57b-4c3a-8b7e-d3d66ef97b41 2025-06-24T16:09:49.481380Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3141s 2025-06-24T16:09:49.481982Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 2) (153.17s) 2025-06-24T16:07:18.093468Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:08:07.581465Z error port forward failed: lost connection to pod 2025-06-24T16:08:07.594840Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward terminated 2025-06-24T16:08:07.635194Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:08:14.848660Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:09:13.640388Z error port forward failed: lost connection to pod 2025-06-24T16:09:14.095195Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward terminated 2025-06-24T16:09:14.131497Z warn pod: echo/server-v1-6f45fcb457-zpzlq port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:09:50.009984Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.91650986s)' === 2025-06-24T16:09:50.010128Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:09:50.010122Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:09:50.213809Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/941e765fc6-6c66-4423-a4f1-3534256a118c 2025-06-24T16:09:50.214216Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2037s 2025-06-24T16:09:50.214655Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:50.459205Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/94eff98bf8-09ac-4218-b903-6a35bb5c6320 2025-06-24T16:09:50.459494Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4490s 2025-06-24T16:09:50.460067Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:09:50.581707Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:09:50Z/94e47a3fd2-f986-4dfb-a981-99f655f3e54b 2025-06-24T16:09:50.581958Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5714s 2025-06-24T16:09:50.582708Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0 (re-run 2) (150.75s) 2025-06-24T16:10:43.690825Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0]' === tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:100: Sending from cluster-0... tracing_test.go:99: retry.UntilSuccessOrFail: timeout while waiting after 50 attempts (last error: cannot get traces from zipkin: zipkin API returned status code: 404) 2025-06-24T16:13:13.691262Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0] (2m30.00043441s)' === 2025-06-24T16:13:13.691376Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T16:13:13.691428Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata/primary-0... 2025-06-24T16:13:13.804721Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/104ee996096-1da0-4254-9ec4-a195f94c05a2 2025-06-24T16:13:13.804904Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1129s 2025-06-24T16:13:13.805543Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:13.949724Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/1048e0d3ab1-e77d-4842-988d-484291935fe9 2025-06-24T16:13:13.950021Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2581s 2025-06-24T16:13:13.950512Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:14.012258Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/10461a66eee-e62b-403c-8da3-2e2afe89635f 2025-06-24T16:13:14.012356Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3204s 2025-06-24T16:13:14.012815Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata (re-run 2) (151.56s) 2025-06-24T16:10:43.680904Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata]' === 2025-06-24T16:13:14.436263Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata] (2m30.755354543s)' === 2025-06-24T16:13:14.436338Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T16:13:14.436341Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider/grpc_exporter_with_initial_metadata... 2025-06-24T16:13:14.561349Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/104f2709652-4389-4e80-bc61-06c91a0e74c9 2025-06-24T16:13:14.561454Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1249s 2025-06-24T16:13:14.562018Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:14.635140Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/10402aca392-aa42-4091-882f-917636eceac9 2025-06-24T16:13:14.635468Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1989s 2025-06-24T16:13:14.635867Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:14.740561Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:10:43Z/10428173ec8-00c0-4228-b998-b1aa5187a67d 2025-06-24T16:13:14.740627Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3040s 2025-06-24T16:13:14.741307Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/tracing/otelcollector TestProxyTracingOpenTelemetryProvider (re-run 2) (152.86s) 2025-06-24T16:10:43.680537Z info tf === BEGIN: Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider]' === tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:11:31.621749Z error port forward failed: lost connection to pod 2025-06-24T16:11:31.864353Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward terminated 2025-06-24T16:11:31.904228Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:11:40.126541Z error port forward failed: lost connection to pod tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:12:37.912958Z error port forward failed: lost connection to pod 2025-06-24T16:12:38.364644Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward terminated 2025-06-24T16:12:38.414360Z warn pod: echo/server-v1-6f45fcb457-76wv8 port forward reconnect success tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig tracing_test.go:112: got traces [] from Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T16:13:15.236794Z info tf === DONE (failed): Test: 'telemetry_tracing_otelcollector[TestProxyTracingOpenTelemetryProvider] (2m31.556254496s)' === 2025-06-24T16:13:15.236866Z error tf === Dumping Namespace echo State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:13:15.236865Z error tf === Dumping Istio Deployment State for TestProxyTracingOpenTelemetryProvider... 2025-06-24T16:13:15.363567Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/105b56be1ae-1e06-4d7b-9c86-ae9f4588f212 2025-06-24T16:13:15.363805Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1267s 2025-06-24T16:13:15.364405Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:15.465671Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/1059b08b3eb-a570-4133-bae2-da9fea991867 2025-06-24T16:13:15.465815Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2287s 2025-06-24T16:13:15.466442Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T16:13:15.565291Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T16:13:15Z/10541f7d8ee-034b-4648-bbb7-bb52cffdc1bc 2025-06-24T16:13:15.565443Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3283s 2025-06-24T16:13:15.565856Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 3 runs, 33 tests, 33 failures in 2297.385s exec: go version