exec: [go test -json -tags=integ -timeout 180m ./tests/integration/telemetry/api -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts -test.skip TestCustomizeMetrics|TestDashboard/pilot-dashboard.json|TestStatsGatewayServerTCPFilter|TestImagePullPolicy/OCI_initial_creation_with_0.0.1|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2,_but_0.0.1_is_already_present_and_policy_is_IfNotPresent|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2,_but_0.0.1_is_already_present_and_policy_is_default|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2._0.0.1_is_already_present_but_policy_is_Always,_so_pull_0.0.2|TestGatewaySelection/OCI_initial_creation_with_latest_for_a_gateway|TestGatewaySelection/OCI_initial_creation_with_latest_for_a_gateway|TestImagePullPolicyWithHTTP] go test pid: 2927233 2025-08-04T17:00:35.804308Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T17:00:35.804382Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T17:00:35.844615Z info tf === Test Framework Settings === 2025-08-04T17:00:35.844716Z info tf TestID: telemetry_api RunID: 0a76d256-3bdf-4e22-bef1-774edaecb0a6 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T17:00:35.844722Z info tf =============================== 2025-08-04T17:00:35.845884Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-0a76d2563bdf4e22b 2025-08-04T17:00:35.846038Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T17:00:35.846053Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T17:00:35.846076Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T17:00:35.846088Z info tf === BEGIN: Building clusters === 2025-08-04T17:00:35.850242Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T17:00:35.850251Z info tf === DONE: Building clusters === 2025-08-04T17:00:35.850278Z info tf === BEGIN: Setup: 'telemetry_api' === 2025-08-04T17:00:35.850298Z info tf === BEGIN: Deploy Istio [Suite=telemetry_api] === 2025-08-04T17:00:35.850304Z info tf === Istio Component Config === 2025-08-04T17:00:35.850333Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.logging.level:xdsproxy:debug,wasm:debug global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T17:00:35.850349Z info tf ================================ 2025-08-04T17:00:35.853474Z info tf skipping deployment as specified in the config 2025-08-04T17:00:35.853533Z info tf === SUCCEEDED: Deploy Istio in 3.225723ms [Suite=telemetry_api]=== 2025-08-04T17:00:36.003219Z info tf === BEGIN: Deploy registry redirector server === 2025-08-04T17:00:36.003342Z info tf === BEGIN: Create namespace registry-redirector === 2025-08-04T17:00:36.088608Z info tf === SUCCEEDED: Create namespace registry-redirector in 85.249448ms === 2025-08-04T17:00:37.872413Z info tf Checking pods ready... 2025-08-04T17:00:37.872482Z info tf Checking pods ready... 2025-08-04T17:00:38.685107Z info tf No pods found... 2025-08-04T17:00:38.886168Z info tf Checking pods ready... 2025-08-04T17:00:38.886239Z info tf Checking pods ready... 2025-08-04T17:00:39.591755Z info tf No pods found... 2025-08-04T17:00:39.992269Z info tf Checking pods ready... 2025-08-04T17:00:39.992366Z info tf Checking pods ready... 2025-08-04T17:00:40.207737Z info tf [ 0] registry-redirector-79c86b49dc-pc94b Pending (Pending) 2025-08-04T17:00:41.008184Z info tf Checking pods ready... 2025-08-04T17:00:41.008237Z info tf Checking pods ready... 2025-08-04T17:00:41.316222Z info tf [ 0] registry-redirector-79c86b49dc-pc94b Pending (Pending) 2025-08-04T17:00:42.917282Z info tf Checking pods ready... 2025-08-04T17:00:42.917377Z info tf Checking pods ready... 2025-08-04T17:00:43.347327Z info tf [ 0] registry-redirector-79c86b49dc-pc94b Pending (Pending) 2025-08-04T17:00:46.548038Z info tf Checking pods ready... 2025-08-04T17:00:46.548094Z info tf Checking pods ready... 2025-08-04T17:00:46.553006Z info tf [ 0] registry-redirector-79c86b49dc-pc94b Running (Ready) 2025-08-04T17:00:46.574344Z info tf registry redirector server address: 172.30.177.159:1338 2025-08-04T17:00:46.864741Z info tf === SUCCEEDED: Deploy registry redirector server === 2025-08-04T17:00:46.864918Z info tf === BEGIN: Create namespace echo === 2025-08-04T17:00:46.864917Z info tf === BEGIN: Create namespace external === 2025-08-04T17:00:46.928041Z info tf === SUCCEEDED: Create namespace external in 63.117621ms === 2025-08-04T17:00:46.947808Z info tf === SUCCEEDED: Create namespace echo in 82.885919ms === 2025-08-04T17:00:47.235962Z info tf === BEGIN: Deploy echo instances === 2025-08-04T17:00:48.802021Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:48.891825Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:48.903075Z info klog Caches are synced controller=echo 2025-08-04T17:00:48.968577Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:48.986971Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:48.992110Z info klog Caches are synced controller=echo 2025-08-04T17:00:48.993188Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:48.997104Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:48.997510Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:49.019559Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:49.019607Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:49.019797Z info klog Waiting for caches to sync controller=echo 2025-08-04T17:00:49.069295Z info klog Caches are synced controller=echo 2025-08-04T17:00:49.087868Z info klog Caches are synced controller=echo 2025-08-04T17:00:49.093675Z info klog Caches are synced controller=echo 2025-08-04T17:00:49.098159Z info klog Caches are synced controller=echo 2025-08-04T17:00:49.098187Z info klog Caches are synced controller=echo 2025-08-04T17:00:49.120729Z info klog Caches are synced controller=echo 2025-08-04T17:00:49.120729Z info klog Caches are synced controller=echo 2025-08-04T17:00:49.120845Z info klog Caches are synced controller=echo 2025-08-04T17:01:00.228054Z info tf === SUCCEEDED: Deploy echo instances in 12.992082213s === 2025-08-04T17:01:00.648799Z info tf Checking pods ready... 2025-08-04T17:01:00.648856Z info tf Checking pods ready... 2025-08-04T17:01:00.655507Z info tf Failed retrieving pods: no matching pod found for selectors: [app.kubernetes.io/name=prometheus] 2025-08-04T17:01:00.856585Z info tf Checking pods ready... 2025-08-04T17:01:00.856632Z info tf Checking pods ready... 2025-08-04T17:01:00.879558Z info tf [ 0] prometheus-858678758b-svnc8 Pending (Pending) 2025-08-04T17:01:01.279947Z info tf Checking pods ready... 2025-08-04T17:01:01.280042Z info tf Checking pods ready... 2025-08-04T17:01:01.284495Z info tf [ 0] prometheus-858678758b-svnc8 Pending (Pending) 2025-08-04T17:01:02.085428Z info tf Checking pods ready... 2025-08-04T17:01:02.085476Z info tf Checking pods ready... 2025-08-04T17:01:02.089800Z info tf [ 0] prometheus-858678758b-svnc8 Pending (Pending) 2025-08-04T17:01:03.690366Z info tf Checking pods ready... 2025-08-04T17:01:03.690429Z info tf Checking pods ready... 2025-08-04T17:01:03.694420Z info tf [ 0] prometheus-858678758b-svnc8 Running (Ready) 2025-08-04T17:01:03.833380Z info tf === DONE: Setup: 'telemetry_api' (27.983080933s) === 2025-08-04T17:01:03.833441Z info tf === BEGIN: Test Run: 'telemetry_api' === PASS tests/integration/telemetry/api.TestAccessLogs/enabled (0.51s) PASS tests/integration/telemetry/api.TestAccessLogs/enabled-with-targetref (9.37s) PASS tests/integration/telemetry/api.TestAccessLogs/disabled (3.82s) PASS tests/integration/telemetry/api.TestAccessLogs (13.69s) PASS tests/integration/telemetry/api.TestAccessLogsFilter (5.08s) PASS tests/integration/telemetry/api.TestAccessLogsMode/client (1.67s) PASS tests/integration/telemetry/api.TestAccessLogsMode/server (11.31s) PASS tests/integration/telemetry/api.TestAccessLogsMode/client-and-server (13.94s) PASS tests/integration/telemetry/api.TestAccessLogsMode (26.92s) PASS tests/integration/telemetry/api.TestAccessLogsDefaultProvider/disabled (6.26s) PASS tests/integration/telemetry/api.TestAccessLogsDefaultProvider/enabled (1.94s) PASS tests/integration/telemetry/api.TestAccessLogsDefaultProvider (8.20s) PASS tests/integration/telemetry/api.TestAccessLogWithFilterState (1.02s) PASS tests/integration/telemetry/api.TestDashboard/istio-mesh-dashboard.json (0.05s) PASS tests/integration/telemetry/api.TestDashboard/istio-service-dashboard.json (0.28s) PASS tests/integration/telemetry/api.TestDashboard/istio-workload-dashboard.json (0.20s) PASS tests/integration/telemetry/api.TestDashboard/istio-performance-dashboard.json (9.58s) PASS tests/integration/telemetry/api.TestDashboard (10.44s) PASS tests/integration/telemetry/api.TestIstioctlMetrics (0.10s) PASS tests/integration/telemetry/api.TestStatsFilter/default (2.07s) === RUN TestStatsFilter/additional-labels 2025-08-04T17:02:11.368736Z info tf === BEGIN: Test: 'telemetry_api[TestStatsFilter/additional-labels]' === stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 8 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 8 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 4 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" 2025-08-04T17:03:05.039456Z error port forward failed: lost connection to pod 2025-08-04T17:03:05.092961Z error port forward failed: lost connection to pod 2025-08-04T17:03:05.443879Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward terminated 2025-08-04T17:03:05.492976Z warn pod: external/external-v1-987785fb7-98ct8 port forward terminated 2025-08-04T17:03:05.519142Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward reconnect success 2025-08-04T17:03:05.575013Z warn pod: external/external-v1-987785fb7-98ct8 port forward reconnect success stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" 2025-08-04T17:03:08.587634Z error port forward failed: lost connection to pod 2025-08-04T17:03:09.011935Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward terminated stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" 2025-08-04T17:03:10.413342Z error port forward failed: lost connection to pod 2025-08-04T17:03:10.840787Z warn pod: echo/statefulset-v1-0 port forward terminated 2025-08-04T17:03:10.925647Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.157741Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.157899Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.353706Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward terminated 2025-08-04T17:03:11.365864Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.428622Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward reconnect success 2025-08-04T17:03:11.575594Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward terminated 2025-08-04T17:03:11.577820Z warn pod: echo/b-v1-759d5cb774-brr5m port forward terminated 2025-08-04T17:03:11.782249Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward terminated 2025-08-04T17:03:12.242149Z error port forward failed: lost connection to pod 2025-08-04T17:03:12.664057Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward terminated 2025-08-04T17:03:13.628803Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward reconnect success 2025-08-04T17:03:13.789444Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward reconnect success 2025-08-04T17:03:13.858502Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward reconnect success 2025-08-04T17:03:13.881648Z warn pod: echo/statefulset-v1-0 port forward reconnect success 2025-08-04T17:03:13.965884Z warn pod: echo/b-v1-759d5cb774-brr5m port forward reconnect success stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" 2025-08-04T17:03:14.983984Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward reconnect success stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:175: test failed: timeout while waiting after 39 attempts (last error: value not found (query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",})) 2025-08-04T17:03:31.391991Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsFilter/additional-labels] (1m20.023249105s)' === 2025-08-04T17:03:31.392104Z error tf === Dumping Namespace external State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.392104Z error tf === Dumping Namespace registry-redirector State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.392153Z error tf === Dumping Istio Deployment State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.392125Z error tf === Dumping Namespace echo State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.704004Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:02:11Z/89f4a5a418-40b1-4eb2-9a00-df7320fb0aab 2025-08-04T17:03:31.704439Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3118s 2025-08-04T17:03:31.704814Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:31.959736Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:02:11Z/89f8c87eaf-5daf-4b96-8968-1acc5f6144eb 2025-08-04T17:03:31.959874Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5672s 2025-08-04T17:03:31.960335Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:32.076396Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:02:11Z/89399a69eb-13d7-4b57-a674-931059e7fdc6 2025-08-04T17:03:32.076585Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6839s 2025-08-04T17:03:32.077091Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestStatsFilter/additional-labels (81.71s) FAIL tests/integration/telemetry/api.TestStatsFilter/additional-labels (81.71s) PASS tests/integration/telemetry/api.TestStatsFilter/mockprom-to-metrics (0.07s) === RUN TestStatsFilter 2025-08-04T17:02:09.294624Z info tf === BEGIN: Test: 'telemetry_api[TestStatsFilter]' === 2025-08-04T17:03:33.146806Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsFilter] (1m23.852180509s)' === 2025-08-04T17:03:33.146861Z error tf === Dumping Namespace external State for TestStatsFilter... 2025-08-04T17:03:33.146870Z error tf === Dumping Istio Deployment State for TestStatsFilter... 2025-08-04T17:03:33.146873Z error tf === Dumping Namespace echo State for TestStatsFilter... 2025-08-04T17:03:33.146866Z error tf === Dumping Namespace registry-redirector State for TestStatsFilter... 2025-08-04T17:03:33.551081Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:33Z/9059d81eb7-d1de-4b81-b6f1-54dfaf08e0c6 2025-08-04T17:03:33.551293Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4042s 2025-08-04T17:03:33.551855Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:33.689530Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:33Z/90ff827858-ebd1-4093-9157-33b8d476bccd 2025-08-04T17:03:33.689674Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5425s 2025-08-04T17:03:33.690224Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:33.866229Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:33Z/90f829255c-32c0-4fd4-890a-04c1d6cdbd93 2025-08-04T17:03:33.866364Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.7192s 2025-08-04T17:03:33.867007Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestStatsFilter (85.69s) FAIL tests/integration/telemetry/api.TestStatsFilter (85.69s) PASS tests/integration/telemetry/api.TestStatsTCPFilter/default (2.04s) === RUN TestStatsTCPFilter/additional-labels 2025-08-04T17:03:37.028214Z info tf === BEGIN: Test: 'telemetry_api[TestStatsTCPFilter/additional-labels]' === stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" 2025-08-04T17:04:11.526523Z error port forward failed: lost connection to pod 2025-08-04T17:04:11.582947Z error port forward failed: lost connection to pod 2025-08-04T17:04:11.944314Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward terminated 2025-08-04T17:04:11.993432Z warn pod: external/external-v1-987785fb7-98ct8 port forward terminated 2025-08-04T17:04:11.998987Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward reconnect success 2025-08-04T17:04:12.059859Z warn pod: external/external-v1-987785fb7-98ct8 port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" 2025-08-04T17:04:17.446447Z error port forward failed: lost connection to pod stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" 2025-08-04T17:04:17.512558Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward terminated 2025-08-04T17:04:17.687709Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" 2025-08-04T17:04:19.645508Z error port forward failed: lost connection to pod 2025-08-04T17:04:19.853801Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward terminated 2025-08-04T17:04:19.867489Z error port forward failed: lost connection to pod 2025-08-04T17:04:19.890692Z error port forward failed: lost connection to pod 2025-08-04T17:04:19.910248Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward reconnect success 2025-08-04T17:04:19.972188Z error port forward failed: lost connection to pod 2025-08-04T17:04:20.077433Z warn pod: echo/b-v1-759d5cb774-brr5m port forward terminated 2025-08-04T17:04:20.156946Z warn pod: echo/b-v1-759d5cb774-brr5m port forward reconnect success 2025-08-04T17:04:20.282693Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward terminated 2025-08-04T17:04:20.340684Z warn pod: echo/statefulset-v1-0 port forward terminated 2025-08-04T17:04:20.354919Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward reconnect success 2025-08-04T17:04:20.399994Z warn pod: echo/statefulset-v1-0 port forward reconnect success 2025-08-04T17:04:20.996847Z error port forward failed: lost connection to pod 2025-08-04T17:04:21.163271Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward terminated 2025-08-04T17:04:21.218887Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 4 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 4 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] 2025-08-04T17:04:39.143799Z error port forward failed: lost connection to pod 2025-08-04T17:04:39.575095Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward terminated 2025-08-04T17:04:39.636028Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:227: test failed: timeout while waiting after 40 attempts (last error: value not found (query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",})) 2025-08-04T17:04:57.044815Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsTCPFilter/additional-labels] (1m20.016595952s)' === 2025-08-04T17:04:57.044896Z error tf === Dumping Namespace external State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.044903Z error tf === Dumping Namespace echo State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.044896Z error tf === Dumping Namespace registry-redirector State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.044945Z error tf === Dumping Istio Deployment State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.323454Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:37Z/9118fa608c-8fdf-4d92-90ff-959b1db65b8f 2025-08-04T17:04:57.323900Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2785s 2025-08-04T17:04:57.324203Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:57.437331Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:37Z/914a65e2c2-e9c1-441c-a9f1-19bf3f5ed221 2025-08-04T17:04:57.437493Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3921s 2025-08-04T17:04:57.438025Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:57.550134Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:37Z/91b91d4f29-96b7-4b01-911f-28c68a9bd997 2025-08-04T17:04:57.550277Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5049s 2025-08-04T17:04:57.550799Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestStatsTCPFilter/additional-labels (81.80s) FAIL tests/integration/telemetry/api.TestStatsTCPFilter/additional-labels (81.80s) === RUN TestStatsTCPFilter 2025-08-04T17:03:34.981517Z info tf === BEGIN: Test: 'telemetry_api[TestStatsTCPFilter]' === 2025-08-04T17:04:58.828192Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsTCPFilter] (1m23.846667564s)' === 2025-08-04T17:04:58.828304Z error tf === Dumping Namespace external State for TestStatsTCPFilter... 2025-08-04T17:04:58.828318Z error tf === Dumping Namespace registry-redirector State for TestStatsTCPFilter... 2025-08-04T17:04:58.828348Z error tf === Dumping Namespace echo State for TestStatsTCPFilter... 2025-08-04T17:04:58.828341Z error tf === Dumping Istio Deployment State for TestStatsTCPFilter... 2025-08-04T17:04:59.132433Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:04:58Z/927bc05b96-252d-47eb-a071-7e27d80406ab 2025-08-04T17:04:59.132556Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3039s 2025-08-04T17:04:59.133078Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:59.276030Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:04:58Z/9207db5672-08b8-4b35-aec9-3e8533dbbc27 2025-08-04T17:04:59.276352Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4477s 2025-08-04T17:04:59.276843Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:59.388149Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:04:58Z/9215441351-c06e-41e3-a61b-56bb12e6b456 2025-08-04T17:04:59.388271Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5596s 2025-08-04T17:04:59.388734Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestStatsTCPFilter (85.70s) FAIL tests/integration/telemetry/api.TestStatsTCPFilter (85.70s) PASS tests/integration/telemetry/api.TestGRPCCountMetrics/istio_request_messages_total (4.20s) PASS tests/integration/telemetry/api.TestGRPCCountMetrics/istio_response_messages_total (0.03s) PASS tests/integration/telemetry/api.TestGRPCCountMetrics (4.23s) PASS tests/integration/telemetry/api.TestImagePullPolicy/Delete_WasmPlugin_wasm-test-module (0.03s) PASS tests/integration/telemetry/api.TestImagePullPolicy/Delete_WasmPlugin_wasm-test-module#01 (0.02s) PASS tests/integration/telemetry/api.TestImagePullPolicy (0.06s) PASS tests/integration/telemetry/api.TestGatewaySelection/Delete_WasmPlugin_wasm-test-module (0.02s) PASS tests/integration/telemetry/api.TestGatewaySelection (0.09s) PASS tests/integration/telemetry/api.TestImagePullPolicyWithHTTP (0.00s) PASS tests/integration/telemetry/api.TestBadWasmRemoteLoad (8.19s) PASS tests/integration/telemetry/api.TestBadWasmWithFailOpen (19.34s) 2025-08-04T17:05:32.591407Z info tf === FAILED: Test Run: 'telemetry_api' (exitCode: 1) === 2025-08-04T17:05:32.591458Z info tf === Suite "telemetry_api" run time: 4m56.741183486s === 2025-08-04T17:05:32.591483Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/trace.yaml 2025-08-04T17:05:32.616596Z error tf === Dumping Namespace echo State for [suite(telemetry_api)]... 2025-08-04T17:05:32.616590Z error tf === Dumping Namespace registry-redirector State for [suite(telemetry_api)]... 2025-08-04T17:05:32.616629Z error tf === Dumping Istio Deployment State for [suite(telemetry_api)]... 2025-08-04T17:05:32.616622Z error tf === Dumping Namespace external State for [suite(telemetry_api)]... 2025-08-04T17:05:33.335827Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:05:32Z/9797f305eb-a16f-4aca-aa74-90ec75097899 2025-08-04T17:05:33.336173Z info tf istioctl ([x internal-debug --all configz]): completed after 0.7191s 2025-08-04T17:05:33.336757Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:05:33.830180Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:05:32Z/972a1f14a1-2fb5-46e6-8f56-a72341735832 2025-08-04T17:05:33.830345Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.2133s 2025-08-04T17:05:33.830897Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:05:34.308098Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:05:32Z/97413fedd8-f94d-4823-afc7-fb0c5fc1cf85 2025-08-04T17:05:34.308283Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.6912s 2025-08-04T17:05:34.308887Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:05:36.908412Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_api] === 2025-08-04T17:05:36.908560Z info tf === SUCCEEDED: Cleanup Istio in 137.357µs [Suite=telemetry_api] === FAIL tests/integration/telemetry/api DONE 37 tests, 4 failures in 301.138s exec: [go test -json -test.run=^TestStatsFilter$/^additional-labels$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/api -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts -test.skip TestCustomizeMetrics|TestDashboard/pilot-dashboard.json|TestStatsGatewayServerTCPFilter|TestImagePullPolicy/OCI_initial_creation_with_0.0.1|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2,_but_0.0.1_is_already_present_and_policy_is_IfNotPresent|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2,_but_0.0.1_is_already_present_and_policy_is_default|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2._0.0.1_is_already_present_but_policy_is_Always,_so_pull_0.0.2|TestGatewaySelection/OCI_initial_creation_with_latest_for_a_gateway|TestGatewaySelection/OCI_initial_creation_with_latest_for_a_gateway|TestImagePullPolicyWithHTTP] go test pid: 3030896 2025-08-04T17:05:48.635167Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T17:05:48.635252Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T17:05:48.674999Z info tf === Test Framework Settings === 2025-08-04T17:05:48.675129Z info tf TestID: telemetry_api RunID: d90f2423-f677-4d90-9b17-95bf0c20e340 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T17:05:48.675137Z info tf =============================== 2025-08-04T17:05:48.675327Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-d90f2423f6774d909 2025-08-04T17:05:48.675434Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T17:05:48.675454Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T17:05:48.675479Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T17:05:48.675492Z info tf === BEGIN: Building clusters === 2025-08-04T17:05:48.681073Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T17:05:48.681086Z info tf === DONE: Building clusters === 2025-08-04T17:05:48.681123Z info tf === BEGIN: Setup: 'telemetry_api' === 2025-08-04T17:05:48.681171Z info tf === BEGIN: Deploy Istio [Suite=telemetry_api] === 2025-08-04T17:05:48.681181Z info tf === Istio Component Config === 2025-08-04T17:05:48.681227Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.logging.level:xdsproxy:debug,wasm:debug global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T17:05:48.681235Z info tf ================================ 2025-08-04T17:05:48.686395Z info tf skipping deployment as specified in the config 2025-08-04T17:05:48.686463Z info tf === SUCCEEDED: Deploy Istio in 5.285216ms [Suite=telemetry_api]=== 2025-08-04T17:05:48.729152Z info tf === BEGIN: Deploy registry redirector server === 2025-08-04T17:05:48.729206Z info tf === BEGIN: Create namespace registry-redirector === 2025-08-04T17:05:48.785344Z info tf === SUCCEEDED: Create namespace registry-redirector in 56.114753ms === 2025-08-04T17:05:49.017341Z info tf Checking pods ready... 2025-08-04T17:05:49.017408Z info tf Checking pods ready... 2025-08-04T17:05:49.032458Z info tf No pods found... 2025-08-04T17:05:49.233160Z info tf Checking pods ready... 2025-08-04T17:05:49.233209Z info tf Checking pods ready... 2025-08-04T17:05:49.239362Z info tf [ 0] registry-redirector-79c86b49dc-9bgd6 Pending (Pending) 2025-08-04T17:05:49.640171Z info tf Checking pods ready... 2025-08-04T17:05:49.640244Z info tf Checking pods ready... 2025-08-04T17:05:49.659950Z info tf [ 0] registry-redirector-79c86b49dc-9bgd6 Pending (Pending) 2025-08-04T17:05:50.460119Z info tf Checking pods ready... 2025-08-04T17:05:50.460211Z info tf Checking pods ready... 2025-08-04T17:05:50.515408Z info tf [ 0] registry-redirector-79c86b49dc-9bgd6 Pending (Pending) 2025-08-04T17:05:52.116091Z info tf Checking pods ready... 2025-08-04T17:05:52.116172Z info tf Checking pods ready... 2025-08-04T17:05:52.120924Z info tf [ 0] registry-redirector-79c86b49dc-9bgd6 Running (container not ready: 'registry-redirector') 2025-08-04T17:05:55.321577Z info tf Checking pods ready... 2025-08-04T17:05:55.321657Z info tf Checking pods ready... 2025-08-04T17:05:55.328804Z info tf [ 0] registry-redirector-79c86b49dc-9bgd6 Running (Ready) 2025-08-04T17:05:55.354690Z info tf registry redirector server address: 172.30.121.87:1338 2025-08-04T17:05:55.468078Z info tf === SUCCEEDED: Deploy registry redirector server === 2025-08-04T17:05:55.468330Z info tf === BEGIN: Create namespace echo === 2025-08-04T17:05:55.468325Z info tf === BEGIN: Create namespace external === 2025-08-04T17:05:55.479064Z info tf === SUCCEEDED: Create namespace echo in 10.746698ms === 2025-08-04T17:05:55.515382Z info tf === SUCCEEDED: Create namespace external in 47.059113ms === 2025-08-04T17:05:55.758985Z info tf === BEGIN: Deploy echo instances === 2025-08-04T17:05:55.811845Z error tf === FAILED: Deploy echo instances === 2025-08-04T17:05:55.811918Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-d90f2423f6774d909/_suite_context/ServiceAccount-Service.3755074222.yaml] to ns echo in cluster cluster-0: apply: patch: serviceaccounts "c" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-08-04T17:05:55.811997Z error tf === Dumping Namespace external State for [suite(telemetry_api)]... 2025-08-04T17:05:55.811997Z error tf === Dumping Istio Deployment State for [suite(telemetry_api)]... 2025-08-04T17:05:55.812096Z error tf === Dumping Namespace registry-redirector State for [suite(telemetry_api)]... 2025-08-04T17:05:55.812114Z error tf === Dumping Namespace echo State for [suite(telemetry_api)]... 2025-08-04T17:05:56.085809Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.116327Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.119937Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:56.144458Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:56.144857Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.154285Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:56.154286Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:56.165787Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:56.187470Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:56.207786Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:56.227762Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.234731Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:56.235175Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:56.302370Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.314111Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:56.405903Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:56.420499Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:56.420509Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.430257Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:56.434967Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:05:55Z/110805c1312-fd3c-4649-8cd4-96e32f7d1b00 2025-08-04T17:05:56.435098Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6227s 2025-08-04T17:05:56.435800Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:05:56.437142Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.441353Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:56.455974Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.478471Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:56.489506Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:56.512830Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:56.533533Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:56.541898Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:56.550804Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:56.571031Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.594624Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:56.636274Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.642255Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:56.642820Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.675415Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:56.681541Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:56.692058Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.713419Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:56.746375Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:56.752554Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:56.791284Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:56.819288Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:56.846380Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:56.856952Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:56.857309Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:56.872394Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.937146Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:56.942620Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:56.949147Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:56.949498Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.958182Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:05:55Z/11099fe745b-9408-4824-8e5e-bc60d9a45ef3 2025-08-04T17:05:56.958318Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1459s 2025-08-04T17:05:56.958891Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:05:56.979714Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:56.983068Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:57.009082Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:57.035648Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:57.050648Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:57.097386Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:57.108561Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:57.147691Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:57.148893Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:57.150851Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:57.154095Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:57.159907Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-08-04T17:05:57.159907Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:57.190369Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-68c69856bc-s878g_echo") 2025-08-04T17:05:57.207443Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:57.252303Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-bd6559c57-w9gkh_echo") 2025-08-04T17:05:57.268304Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:57.278347Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-7fb68cd64-9nwl2_echo") 2025-08-04T17:05:57.282927Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-7c8894f69-d7lsz_echo") 2025-08-04T17:05:57.301239Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("mock-prom-v1-55fc7dcf94-7wtkm_echo") 2025-08-04T17:05:57.312554Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo")) 2025-08-04T17:05:57.327138Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:05:55Z/1104937cfe9-c9c4-4897-9350-570eea711590 2025-08-04T17:05:57.327280Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.5148s 2025-08-04T17:05:57.327864Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:05:57.382303Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-66c64b674f-rlgcw_echo") 2025-08-04T17:05:57.393011Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo")) 2025-08-04T17:05:57.954295Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-d90f2423f6774d909/_suite_context/ServiceAccount-Service.3755074222.yaml] to ns echo in cluster cluster-0: apply: patch: serviceaccounts "c" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-08-04T17:05:57.954353Z info tf === FAILED: Setup: 'telemetry_api' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-d90f2423f6774d909/_suite_context/ServiceAccount-Service.3755074222.yaml] to ns echo in cluster cluster-0: apply: patch: serviceaccounts "c" is forbidden: unable to create new content in namespace echo because it is being terminated) === 2025-08-04T17:05:57.954365Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-d90f2423f6774d909/_suite_context/ServiceAccount-Service.3755074222.yaml] to ns echo in cluster cluster-0: apply: patch: serviceaccounts "c" is forbidden: unable to create new content in namespace echo because it is being terminated 2025-08-04T17:05:57.954460Z error tf === Dumping Istio Deployment State for [suite(telemetry_api)]... 2025-08-04T17:05:57.954437Z error tf === Dumping Namespace registry-redirector State for [suite(telemetry_api)]... 2025-08-04T17:05:57.954437Z error tf === Dumping Namespace external State for [suite(telemetry_api)]... 2025-08-04T17:05:57.954478Z error tf === Dumping Namespace echo State for [suite(telemetry_api)]... 2025-08-04T17:05:58.047513Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/sotw-v1-7fb68cd64-9nwl2/istio-proxy for: pods "sotw-v1-7fb68cd64-9nwl2" not found 2025-08-04T17:05:58.061673Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo/sotw-v1-7fb68cd64-9nwl2/istio-validation for: pods "sotw-v1-7fb68cd64-9nwl2" not found 2025-08-04T17:05:58.141879Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-7fb68cd64-9nwl2" not found in the "echo" namespace) 2025-08-04T17:05:58.146329Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-7fb68cd64-9nwl2" not found in the "echo" namespace) 2025-08-04T17:05:58.199220Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:05:55Z/110e9b1a862-da13-4b9f-bcae-e25184b459c7 2025-08-04T17:05:58.199564Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2448s 2025-08-04T17:05:58.200249Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:05:58.347050Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_api] === 2025-08-04T17:05:58.347153Z info tf === SUCCEEDED: Cleanup Istio in 94.431µs [Suite=telemetry_api] === FAIL tests/integration/telemetry/api exec: [go test -json -test.run=^TestStatsTCPFilter$/^additional-labels$ -tags=integ -timeout 180m istio.io/istio/tests/integration/telemetry/api -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts -test.skip TestCustomizeMetrics|TestDashboard/pilot-dashboard.json|TestStatsGatewayServerTCPFilter|TestImagePullPolicy/OCI_initial_creation_with_0.0.1|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2,_but_0.0.1_is_already_present_and_policy_is_IfNotPresent|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2,_but_0.0.1_is_already_present_and_policy_is_default|TestImagePullPolicy/OCI_upstream_is_upgraded_to_0.0.2._0.0.1_is_already_present_but_policy_is_Always,_so_pull_0.0.2|TestGatewaySelection/OCI_initial_creation_with_latest_for_a_gateway|TestGatewaySelection/OCI_initial_creation_with_latest_for_a_gateway|TestImagePullPolicyWithHTTP] go test pid: 3032187 2025-08-04T17:06:10.764707Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T17:06:10.764769Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T17:06:10.808371Z info tf === Test Framework Settings === 2025-08-04T17:06:10.808574Z info tf TestID: telemetry_api RunID: 63ec0b82-f49c-4656-9810-0ad9c2c2e3c1 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T17:06:10.808586Z info tf =============================== 2025-08-04T17:06:10.808855Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-63ec0b82f49c46569 2025-08-04T17:06:10.809001Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T17:06:10.809020Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T17:06:10.809045Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T17:06:10.809056Z info tf === BEGIN: Building clusters === 2025-08-04T17:06:10.816415Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T17:06:10.816458Z info tf === DONE: Building clusters === 2025-08-04T17:06:10.816490Z info tf === BEGIN: Setup: 'telemetry_api' === 2025-08-04T17:06:10.816519Z info tf === BEGIN: Deploy Istio [Suite=telemetry_api] === 2025-08-04T17:06:10.816524Z info tf === Istio Component Config === 2025-08-04T17:06:10.816573Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.logging.level:xdsproxy:debug,wasm:debug global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T17:06:10.816579Z info tf ================================ 2025-08-04T17:06:10.819153Z info tf skipping deployment as specified in the config 2025-08-04T17:06:10.819168Z info tf === SUCCEEDED: Deploy Istio in 2.642764ms [Suite=telemetry_api]=== 2025-08-04T17:06:10.845831Z info tf === BEGIN: Deploy registry redirector server === 2025-08-04T17:06:10.845898Z info tf === BEGIN: Create namespace registry-redirector === 2025-08-04T17:06:10.856524Z info tf === SUCCEEDED: Create namespace registry-redirector in 10.596063ms === 2025-08-04T17:06:10.862841Z info tf === SUCCEEDED: Deploy registry redirector server === 2025-08-04T17:06:10.862906Z error tf === Dumping Namespace registry-redirector State for [suite(telemetry_api)]... 2025-08-04T17:06:10.862913Z error tf === Dumping Istio Deployment State for [suite(telemetry_api)]... 2025-08-04T17:06:10.870884Z error tf Test setup error: failed to apply rendered /root/mohit/istio/pkg/test/framework/components/registryredirector/registry_redirector_server.yaml, err: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-63ec0b82f49c46569/_suite_context/Deployment-Service.4212539383.yaml] to ns registry-redirector in cluster cluster-0: apply: patch: services "registry-redirector" is forbidden: unable to create new content in namespace registry-redirector because it is being terminated 2025-08-04T17:06:10.870986Z info tf === FAILED: Setup: 'telemetry_api' (failed to apply rendered /root/mohit/istio/pkg/test/framework/components/registryredirector/registry_redirector_server.yaml, err: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-63ec0b82f49c46569/_suite_context/Deployment-Service.4212539383.yaml] to ns registry-redirector in cluster cluster-0: apply: patch: services "registry-redirector" is forbidden: unable to create new content in namespace registry-redirector because it is being terminated) === 2025-08-04T17:06:10.870996Z error tf Exiting due to setup failure: failed to apply rendered /root/mohit/istio/pkg/test/framework/components/registryredirector/registry_redirector_server.yaml, err: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/telemetry-api/artifacts/telemetry-api-63ec0b82f49c46569/_suite_context/Deployment-Service.4212539383.yaml] to ns registry-redirector in cluster cluster-0: apply: patch: services "registry-redirector" is forbidden: unable to create new content in namespace registry-redirector because it is being terminated 2025-08-04T17:06:10.871197Z error tf === Dumping Namespace registry-redirector State for [suite(telemetry_api)]... 2025-08-04T17:06:10.871251Z error tf === Dumping Istio Deployment State for [suite(telemetry_api)]... 2025-08-04T17:06:10.942370Z info tf === BEGIN: Cleanup Istio [Suite=telemetry_api] === 2025-08-04T17:06:10.942450Z info tf === SUCCEEDED: Cleanup Istio in 70.75µs [Suite=telemetry_api] === FAIL tests/integration/telemetry/api === Failed === FAIL: tests/integration/telemetry/api TestStatsFilter/additional-labels (81.71s) 2025-08-04T17:02:11.368736Z info tf === BEGIN: Test: 'telemetry_api[TestStatsFilter/additional-labels]' === stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 12 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 5 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 8 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 9 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 10 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 11 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_version", wanted "v1" but got "unknown" stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 7 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: Series 8 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: query "istio_requests_total" returned 9 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 2 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 4 (source: istio-system/istio-ingressgateway-b7b89b8d5-prhjr) stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 5 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 6 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: Series 7 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "418" stats_test.go:142: missing labels: [upstream_custom_label source_version] stats_test.go:142: Series 8 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "source_workload", wanted "a-v1" but got "istio-ingressgateway" stats_test.go:142: for label "source_workload_namespace", wanted "echo" but got "istio-system" stats_test.go:142: for label "source_app", wanted "a" but got "istio-ingressgateway" stats_test.go:142: for label "response_code", wanted "200" but got "520" stats_test.go:142: missing labels: [source_version upstream_custom_label] stats_test.go:142: query "istio_requests_total" returned 4 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" 2025-08-04T17:03:05.039456Z error port forward failed: lost connection to pod 2025-08-04T17:03:05.092961Z error port forward failed: lost connection to pod 2025-08-04T17:03:05.443879Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward terminated 2025-08-04T17:03:05.492976Z warn pod: external/external-v1-987785fb7-98ct8 port forward terminated 2025-08-04T17:03:05.519142Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward reconnect success 2025-08-04T17:03:05.575013Z warn pod: external/external-v1-987785fb7-98ct8 port forward reconnect success stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" 2025-08-04T17:03:08.587634Z error port forward failed: lost connection to pod 2025-08-04T17:03:09.011935Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward terminated stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" 2025-08-04T17:03:10.413342Z error port forward failed: lost connection to pod 2025-08-04T17:03:10.840787Z warn pod: echo/statefulset-v1-0 port forward terminated 2025-08-04T17:03:10.925647Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.157741Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.157899Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.353706Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward terminated 2025-08-04T17:03:11.365864Z error port forward failed: lost connection to pod 2025-08-04T17:03:11.428622Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward reconnect success 2025-08-04T17:03:11.575594Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward terminated 2025-08-04T17:03:11.577820Z warn pod: echo/b-v1-759d5cb774-brr5m port forward terminated 2025-08-04T17:03:11.782249Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward terminated 2025-08-04T17:03:12.242149Z error port forward failed: lost connection to pod 2025-08-04T17:03:12.664057Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward terminated 2025-08-04T17:03:13.628803Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward reconnect success 2025-08-04T17:03:13.789444Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward reconnect success 2025-08-04T17:03:13.858502Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward reconnect success 2025-08-04T17:03:13.881648Z warn pod: echo/statefulset-v1-0 port forward reconnect success 2025-08-04T17:03:13.965884Z warn pod: echo/b-v1-759d5cb774-brr5m port forward reconnect success stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" 2025-08-04T17:03:14.983984Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward reconnect success stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: query "istio_requests_total" returned 3 series, but none matched our query exactly. stats_test.go:142: Original query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",} stats_test.go:142: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:142: for label "reporter", wanted "source" but got "destination" stats_test.go:142: missing labels: [upstream_custom_label] stats_test.go:142: Series 2 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:142: for label "destination_app", wanted "b" but got "unknown" stats_test.go:142: for label "upstream_custom_label", wanted "b" but got "unknown" stats_test.go:142: for label "destination_version", wanted "v1" but got "unknown" stats_test.go:142: for label "destination_service_name", wanted "b" but got "naked" stats_test.go:142: for label "destination_service", wanted "b.echo.svc.cluster.local" but got "naked.echo.svc.cluster.local" stats_test.go:175: test failed: timeout while waiting after 39 attempts (last error: value not found (query: istio_requests_total{destination_app="b",destination_service="b.echo.svc.cluster.local",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",reporter="source",request_protocol="http",response_code="200",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",upstream_custom_label="b",})) 2025-08-04T17:03:31.391991Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsFilter/additional-labels] (1m20.023249105s)' === 2025-08-04T17:03:31.392104Z error tf === Dumping Namespace external State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.392104Z error tf === Dumping Namespace registry-redirector State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.392153Z error tf === Dumping Istio Deployment State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.392125Z error tf === Dumping Namespace echo State for TestStatsFilter/additional-labels... 2025-08-04T17:03:31.704004Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:02:11Z/89f4a5a418-40b1-4eb2-9a00-df7320fb0aab 2025-08-04T17:03:31.704439Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3118s 2025-08-04T17:03:31.704814Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:31.959736Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:02:11Z/89f8c87eaf-5daf-4b96-8968-1acc5f6144eb 2025-08-04T17:03:31.959874Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5672s 2025-08-04T17:03:31.960335Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:32.076396Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:02:11Z/89399a69eb-13d7-4b57-a674-931059e7fdc6 2025-08-04T17:03:32.076585Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6839s 2025-08-04T17:03:32.077091Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/api TestStatsFilter (85.69s) 2025-08-04T17:02:09.294624Z info tf === BEGIN: Test: 'telemetry_api[TestStatsFilter]' === 2025-08-04T17:03:33.146806Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsFilter] (1m23.852180509s)' === 2025-08-04T17:03:33.146861Z error tf === Dumping Namespace external State for TestStatsFilter... 2025-08-04T17:03:33.146870Z error tf === Dumping Istio Deployment State for TestStatsFilter... 2025-08-04T17:03:33.146873Z error tf === Dumping Namespace echo State for TestStatsFilter... 2025-08-04T17:03:33.146866Z error tf === Dumping Namespace registry-redirector State for TestStatsFilter... 2025-08-04T17:03:33.551081Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:33Z/9059d81eb7-d1de-4b81-b6f1-54dfaf08e0c6 2025-08-04T17:03:33.551293Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4042s 2025-08-04T17:03:33.551855Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:33.689530Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:33Z/90ff827858-ebd1-4093-9157-33b8d476bccd 2025-08-04T17:03:33.689674Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5425s 2025-08-04T17:03:33.690224Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:03:33.866229Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:33Z/90f829255c-32c0-4fd4-890a-04c1d6cdbd93 2025-08-04T17:03:33.866364Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.7192s 2025-08-04T17:03:33.867007Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/api TestStatsTCPFilter/additional-labels (81.80s) 2025-08-04T17:03:37.028214Z info tf === BEGIN: Test: 'telemetry_api[TestStatsTCPFilter/additional-labels]' === stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" 2025-08-04T17:04:11.526523Z error port forward failed: lost connection to pod 2025-08-04T17:04:11.582947Z error port forward failed: lost connection to pod 2025-08-04T17:04:11.944314Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward terminated 2025-08-04T17:04:11.993432Z warn pod: external/external-v1-987785fb7-98ct8 port forward terminated 2025-08-04T17:04:11.998987Z warn pod: echo/naked-v1-84758944d9-hd7hz port forward reconnect success 2025-08-04T17:04:12.059859Z warn pod: external/external-v1-987785fb7-98ct8 port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" 2025-08-04T17:04:17.446447Z error port forward failed: lost connection to pod stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" 2025-08-04T17:04:17.512558Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward terminated 2025-08-04T17:04:17.687709Z warn pod: echo/proxyless-grpc-v1-7768d575bc-9595l port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 5 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 4 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "source_workload", wanted "a-v1" but got "unknown" stats_test.go:214: for label "source_app", wanted "a" but got "unknown" stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: for label "source_cluster", wanted "Kubernetes" but got "unknown" stats_test.go:214: for label "destination_service_name", wanted "b" but got "InboundPassthroughCluster" stats_test.go:214: for label "source_version", wanted "v1" but got "unknown" 2025-08-04T17:04:19.645508Z error port forward failed: lost connection to pod 2025-08-04T17:04:19.853801Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward terminated 2025-08-04T17:04:19.867489Z error port forward failed: lost connection to pod 2025-08-04T17:04:19.890692Z error port forward failed: lost connection to pod 2025-08-04T17:04:19.910248Z warn pod: echo/c-v1-66c64b674f-rlgcw port forward reconnect success 2025-08-04T17:04:19.972188Z error port forward failed: lost connection to pod 2025-08-04T17:04:20.077433Z warn pod: echo/b-v1-759d5cb774-brr5m port forward terminated 2025-08-04T17:04:20.156946Z warn pod: echo/b-v1-759d5cb774-brr5m port forward reconnect success 2025-08-04T17:04:20.282693Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward terminated 2025-08-04T17:04:20.340684Z warn pod: echo/statefulset-v1-0 port forward terminated 2025-08-04T17:04:20.354919Z warn pod: echo/headless-v1-7c8894f69-d7lsz port forward reconnect success 2025-08-04T17:04:20.399994Z warn pod: echo/statefulset-v1-0 port forward reconnect success 2025-08-04T17:04:20.996847Z error port forward failed: lost connection to pod 2025-08-04T17:04:21.163271Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward terminated 2025-08-04T17:04:21.218887Z warn pod: echo/sotw-v1-7fb68cd64-9nwl2 port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 4 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 4 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 1 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 2 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: Series 3 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] 2025-08-04T17:04:39.143799Z error port forward failed: lost connection to pod 2025-08-04T17:04:39.575095Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward terminated 2025-08-04T17:04:39.636028Z warn pod: echo/mock-prom-v1-55fc7dcf94-7wtkm port forward reconnect success stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:214: query "istio_tcp_connections_opened_total" returned 2 series, but none matched our query exactly. stats_test.go:214: Original query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",} stats_test.go:214: Series 0 (source: echo/a-v1-68c69856bc-s878g) stats_test.go:214: for label "downstream_custom_label", wanted "a" but got "unknown" stats_test.go:214: Series 1 (source: echo/b-v1-759d5cb774-brr5m) stats_test.go:214: for label "reporter", wanted "destination" but got "source" stats_test.go:214: missing labels: [downstream_custom_label] stats_test.go:227: test failed: timeout while waiting after 40 attempts (last error: value not found (query: istio_tcp_connections_opened_total{destination_app="b",destination_canonical_revision="v1",destination_canonical_service="b",destination_service_name="b",destination_service_namespace="echo",destination_version="v1",destination_workload_namespace="echo",downstream_custom_label="a",reporter="destination",request_protocol="tcp",source_app="a",source_cluster="Kubernetes",source_version="v1",source_workload="a-v1",source_workload_namespace="echo",})) 2025-08-04T17:04:57.044815Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsTCPFilter/additional-labels] (1m20.016595952s)' === 2025-08-04T17:04:57.044896Z error tf === Dumping Namespace external State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.044903Z error tf === Dumping Namespace echo State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.044896Z error tf === Dumping Namespace registry-redirector State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.044945Z error tf === Dumping Istio Deployment State for TestStatsTCPFilter/additional-labels... 2025-08-04T17:04:57.323454Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:37Z/9118fa608c-8fdf-4d92-90ff-959b1db65b8f 2025-08-04T17:04:57.323900Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2785s 2025-08-04T17:04:57.324203Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:57.437331Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:37Z/914a65e2c2-e9c1-441c-a9f1-19bf3f5ed221 2025-08-04T17:04:57.437493Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3921s 2025-08-04T17:04:57.438025Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:57.550134Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:03:37Z/91b91d4f29-96b7-4b01-911f-28c68a9bd997 2025-08-04T17:04:57.550277Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5049s 2025-08-04T17:04:57.550799Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/telemetry/api TestStatsTCPFilter (85.70s) 2025-08-04T17:03:34.981517Z info tf === BEGIN: Test: 'telemetry_api[TestStatsTCPFilter]' === 2025-08-04T17:04:58.828192Z info tf === DONE (failed): Test: 'telemetry_api[TestStatsTCPFilter] (1m23.846667564s)' === 2025-08-04T17:04:58.828304Z error tf === Dumping Namespace external State for TestStatsTCPFilter... 2025-08-04T17:04:58.828318Z error tf === Dumping Namespace registry-redirector State for TestStatsTCPFilter... 2025-08-04T17:04:58.828348Z error tf === Dumping Namespace echo State for TestStatsTCPFilter... 2025-08-04T17:04:58.828341Z error tf === Dumping Istio Deployment State for TestStatsTCPFilter... 2025-08-04T17:04:59.132433Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:04:58Z/927bc05b96-252d-47eb-a071-7e27d80406ab 2025-08-04T17:04:59.132556Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3039s 2025-08-04T17:04:59.133078Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:59.276030Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:04:58Z/9207db5672-08b8-4b35-aec9-3e8533dbbc27 2025-08-04T17:04:59.276352Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4477s 2025-08-04T17:04:59.276843Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T17:04:59.388149Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T17:04:58Z/9215441351-c06e-41e3-a61b-56bb12e6b456 2025-08-04T17:04:59.388271Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5596s 2025-08-04T17:04:59.388734Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 2 runs, 37 tests, 4 failures in 335.171s exec: go version