exec: [go test -json -tags=integ -timeout 180m ./tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3421322 2025-06-24T10:13:14.830618Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:13:14.830679Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:13:14.868523Z info tf === Test Framework Settings === 2025-06-24T10:13:14.868651Z info tf TestID: security_ca_custom_root RunID: 0c787312-767f-4103-901c-aac2be792661 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:13:14.868661Z info tf =============================== 2025-06-24T10:13:14.870256Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-0c78731 2025-06-24T10:13:14.870404Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:13:14.870426Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:13:14.870454Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:13:14.870466Z info tf === BEGIN: Building clusters === 2025-06-24T10:13:14.876542Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:13:14.876596Z info tf === DONE: Building clusters === 2025-06-24T10:13:14.876618Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:13:14.907678Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:13:14.907705Z info tf === SUCCESS: context setup function === 2025-06-24T10:13:14.907714Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:13:14.907731Z info tf === Istio Component Config === 2025-06-24T10:13:14.907825Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:13:14.907832Z info tf ================================ 2025-06-24T10:13:14.911274Z info tf skipping deployment as specified in the config 2025-06-24T10:13:14.911287Z info tf === SUCCEEDED: Deploy Istio in 3.571793ms [Suite=security_ca_custom_root]=== 2025-06-24T10:13:14.911293Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:13:15.003852Z info tf === SUCCEEDED: Create namespace echo1 in 92.554304ms === Generating RSA private key, 2048 bit long modulus (2 primes) *****************************************************************+++++ ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-0c78731/_suite_context/ca-custom-root-253099263/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***************+++++ ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-0c78731/_suite_context/ca-custom-root-253099263/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *****************************************************************************+++++ **************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-0c78731/_suite_context/ca-custom-root-253099263/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:13:16.219216Z info tf === BEGIN: Create namespace external === 2025-06-24T10:13:16.236817Z info tf === SUCCEEDED: Create namespace external in 17.626475ms === 2025-06-24T10:13:16.351253Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:13:17.623588Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.667166Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.677841Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.678503Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.679138Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.684492Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.724217Z info klog Caches are synced for echo 2025-06-24T10:13:17.749751Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.749868Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.749976Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.754323Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.757726Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.758308Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.767283Z info klog Caches are synced for echo 2025-06-24T10:13:17.778715Z info klog Caches are synced for echo 2025-06-24T10:13:17.778741Z info klog Caches are synced for echo 2025-06-24T10:13:17.780205Z info klog Caches are synced for echo 2025-06-24T10:13:17.784929Z info klog Caches are synced for echo 2025-06-24T10:13:17.847029Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.849979Z info klog Caches are synced for echo 2025-06-24T10:13:17.850008Z info klog Caches are synced for echo 2025-06-24T10:13:17.850049Z info klog Caches are synced for echo 2025-06-24T10:13:17.853122Z info klog Waiting for caches to sync for echo 2025-06-24T10:13:17.854401Z info klog Caches are synced for echo 2025-06-24T10:13:17.858074Z info klog Caches are synced for echo 2025-06-24T10:13:17.859358Z info klog Caches are synced for echo 2025-06-24T10:13:17.948213Z info klog Caches are synced for echo 2025-06-24T10:13:17.953207Z info klog Caches are synced for echo 2025-06-24T10:13:46.571074Z info tf === SUCCEEDED: Deploy echo instances in 30.219813284s === 2025-06-24T10:13:46.603961Z info tf === DONE: Setup: 'security_ca_custom_root' (31.727332026s) === 2025-06-24T10:13:46.603987Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === PASS tests/integration/security/ca_custom_root.TestMultiRootSetup/From_primary-0/server:server-naked-foo-alt[success] (5.35s) PASS tests/integration/security/ca_custom_root.TestMultiRootSetup/From_primary-0 (5.35s) PASS tests/integration/security/ca_custom_root.TestMultiRootSetup (5.41s) === RUN TestSecureNaming 2025-06-24T10:13:52.017435Z info tf === BEGIN: Test: 'security_ca_custom_root[TestSecureNaming]' === secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:114: retry.UntilSuccessOrFail: timeout while waiting after 10 attempts (last error: CA certificate (-----BEGIN CERTIFICATE----- MIIC/TCCAeWgAwIBAgIRAOultIFQpmMoQ3Ba4vBy3uowDQYJKoZIhvcNAQELBQAw GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yNTA2MjMxNzUyMTFaFw0zNTA2 MjExNzUyMTFaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3 DQEBAQUAA4IBDwAwggEKAoIBAQDD61K6//kQ50joJsVYbHd4cHQMVhi0HNh4K71M VAuIRgtcV/1g5lsTSzlytjzYSgW0VTAP1VFBQqNVUo7pmQNLEZaWHTpi/8/I5N5D 5/hztT8WqCCaWoyK2nh+fQX+FDzPTI/O8nwIx/FRGXflx2rIpiDrB3kX3ERV5l9d tKsXMIbHVJQ7xn8c24As0kdBzZKmzbHmzaLgpNHftqPiuD0PYLhwCHvQvIXLapuW I3LvuPoRGtnxPTSh8B4wnRz9LEfjmDVJZZ89jDN1Qubj3n4OsSLUTLsJ5bFgeTmO 7hghz4zM8zM6ZAY5zqFmw2zcmAlFrACDKtIMf6ONd71BArkFAgMBAAGjQjBAMA4G A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT0Czg2lEDA pSVzoN5aJDxtUzmFwzANBgkqhkiG9w0BAQsFAAOCAQEAvSGwExBY/OLSdjv10G0e T58ioViIHqsCDR8JC23xJstoHg/RGQbGZsjiTIAv6uifa3uOWmfO2bhoZTeBtAMy R/LhgKQuukZltVRoLyp7bCALB0ZLNoPrbrgR83SEWtoCEQ9lW7iT+f/MKIMt9qja g0dURbkIAasOF58uTSgSOwEJw2M5kmq6//K/krVWM44FjsMdwgyDw5s6EGRU/jB8 6OK8czEjuPLjAMbYIQKTaorEt5zu9PFFJyLTAw8fw8SXoqDgS6/1zzWdv3XOwRR5 dIv2JC9JiGi6ch9ICpItdK9pUw4iouKfUw/opsUHjayU8FVqWtzVxZkjJwIVdZz1 1A== -----END CERTIFICATE----- ) not matching plugin cert (-----BEGIN CERTIFICATE----- MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN 8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a 3g== -----END CERTIFICATE----- )) 2025-06-24T10:14:02.026331Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestSecureNaming] (10.008893754s)' === 2025-06-24T10:14:02.026426Z error tf === Dumping Namespace external State for TestSecureNaming... 2025-06-24T10:14:02.026421Z error tf === Dumping Istio Deployment State for TestSecureNaming... 2025-06-24T10:14:02.026451Z error tf === Dumping Namespace echo1 State for TestSecureNaming... 2025-06-24T10:14:02.338526Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:13:52Z/16187d2201-0932-442a-a2db-eff7aab8b168 2025-06-24T10:14:02.338766Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3120s 2025-06-24T10:14:02.339390Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:02.454251Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:13:52Z/16b1be7fae-3ec2-44c3-aacd-7bad21bcca72 2025-06-24T10:14:02.454481Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4277s 2025-06-24T10:14:02.455275Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:02.531891Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:13:52Z/1600f3c1f6-a63e-4dcb-9787-6ab12681f313 2025-06-24T10:14:02.532117Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5053s 2025-06-24T10:14:02.532697Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:03.345047Z error tf === Dumping Namespace istio-system State for TestSecureNaming... --- FAIL: TestSecureNaming (11.87s) FAIL tests/integration/security/ca_custom_root.TestSecureNaming (11.87s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] 2025-06-24T10:14:03.901376Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === 2025-06-24T10:14:28.009338Z error port forward failed: lost connection to pod 2025-06-24T10:14:28.317020Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward terminated 2025-06-24T10:14:28.355160Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward reconnect success 2025-06-24T10:14:28.747815Z error port forward failed: lost connection to pod 2025-06-24T10:14:28.762393Z error port forward failed: lost connection to pod 2025-06-24T10:14:28.782842Z warn pod: echo1/naked-v1-757556fc5c-fx9tx port forward terminated 2025-06-24T10:14:28.818217Z warn pod: echo1/naked-v1-757556fc5c-fx9tx port forward reconnect success 2025-06-24T10:14:28.885331Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward terminated 2025-06-24T10:14:28.923556Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward reconnect success 2025-06-24T10:14:29.654778Z error port forward failed: lost connection to pod 2025-06-24T10:14:29.846349Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward terminated 2025-06-24T10:14:29.882781Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward reconnect success 2025-06-24T10:14:30.289606Z error port forward failed: lost connection to pod 2025-06-24T10:14:30.453789Z warn pod: external/external-v1-f9d74566f-lg7nm port forward terminated 2025-06-24T10:14:30.508342Z warn pod: external/external-v1-f9d74566f-lg7nm port forward reconnect success test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.3.45:48546->172.30.13.26:443: read: connection reset by peer 2025-06-24T10:14:33.902412Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.001025957s)' === 2025-06-24T10:14:33.902492Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:14:33.902550Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:14:33.902520Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:14:36.150952Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/178e6e3d25-c3e5-465e-8948-c624f92f6b20 2025-06-24T10:14:36.151163Z info tf istioctl ([x internal-debug --all mcsz]): completed after 2.2481s 2025-06-24T10:14:36.151753Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:36.251937Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17e1e447b2-03a0-4675-a65d-bfa699ca8e86 2025-06-24T10:14:36.252218Z info tf istioctl ([x internal-debug --all configz]): completed after 2.3491s 2025-06-24T10:14:36.252695Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:36.367824Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17f0ea4a98-ac41-4245-bbaa-b3ca4f977bf8 2025-06-24T10:14:36.368020Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.4649s 2025-06-24T10:14:36.368674Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (33.16s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (33.16s) PASS tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-bar[fail] (0.05s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0 2025-06-24T10:14:03.901130Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-06-24T10:14:37.116500Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (33.215370633s)' === 2025-06-24T10:14:37.116538Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:14:37.116562Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:14:37.116535Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:14:37.296402Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17555f2755-71e0-44fe-a7b8-4c082581d274 2025-06-24T10:14:37.296541Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1796s 2025-06-24T10:14:37.297000Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:37.408221Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17e73a9598-ad8f-473d-82c2-3591b833d397 2025-06-24T10:14:37.408422Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2915s 2025-06-24T10:14:37.409072Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:37.480181Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17d58e1f36-c692-4992-b075-9ec5b69bb963 2025-06-24T10:14:37.480323Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3634s 2025-06-24T10:14:37.480983Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0 (34.03s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0 (34.03s) === RUN TestTrustDomainAliasSecureNaming 2025-06-24T10:14:03.882807Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-06-24T10:14:37.929664Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (34.046845913s)' === 2025-06-24T10:14:37.929761Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:14:37.929758Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:14:37.929782Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:14:38.122609Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/178f00552d-fc78-4c1f-94d2-47143a704abb 2025-06-24T10:14:38.122796Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1928s 2025-06-24T10:14:38.123525Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:38.250687Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17dabebecb-83a3-48d1-80e9-04dd1a0f2388 2025-06-24T10:14:38.251014Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3210s 2025-06-24T10:14:38.251651Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:38.319501Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17092a6082-d8c0-41cd-9e3f-eeb98b91b9a4 2025-06-24T10:14:38.319585Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3896s 2025-06-24T10:14:38.320120Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming (34.89s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming (34.89s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:http-plaintext[allow] (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:tcp-plaintext[allow] (0.03s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:http-plaintext[allow] (5.26s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:tcp-plaintext[allow] (0.01s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:http-mtls[allow] (0.05s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:tcp-mtls[allow] (0.03s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:http-mtls[deny] (0.06s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls[deny] (0.07s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls-pass-through[deny] (0.05s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-06-24T10:14:44.408356Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === 2025-06-24T10:14:49.092340Z error port forward failed: lost connection to pod 2025-06-24T10:14:49.531962Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward terminated 2025-06-24T10:14:49.568985Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward reconnect success 2025-06-24T10:14:49.734066Z error port forward failed: lost connection to pod 2025-06-24T10:14:49.784788Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward terminated 2025-06-24T10:14:49.820394Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward reconnect success 2025-06-24T10:14:50.103340Z error port forward failed: lost connection to pod 2025-06-24T10:14:50.146016Z error port forward failed: lost connection to pod 2025-06-24T10:14:50.595912Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward terminated 2025-06-24T10:14:50.600152Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward terminated 2025-06-24T10:14:50.636416Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward reconnect success 2025-06-24T10:14:50.637159Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward reconnect success 2025-06-24T10:14:50.711747Z error port forward failed: lost connection to pod 2025-06-24T10:14:51.157577Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward terminated 2025-06-24T10:14:51.189845Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward reconnect success 2025-06-24T10:14:51.561966Z error port forward failed: lost connection to pod 2025-06-24T10:14:52.006049Z error port forward failed: lost connection to pod 2025-06-24T10:14:52.045847Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:14:52.081007Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:14:52.322224Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward terminated 2025-06-24T10:14:52.359471Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:15:14.409532Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.001171928s)' === 2025-06-24T10:15:14.409630Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:15:14.409651Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:15:14.409686Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:15:14.630081Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18dc276bcc-8596-43ac-98de-fd7c4ee5bd5b 2025-06-24T10:15:14.630376Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2203s 2025-06-24T10:15:14.631060Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:14.726373Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/1840095c63-a037-4787-af51-3ab478e7210a 2025-06-24T10:15:14.726582Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3165s 2025-06-24T10:15:14.727349Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:14.797759Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/183fe7cf00-68e3-49d4-a74a-0c718d02e9a8 2025-06-24T10:15:14.797928Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3878s 2025-06-24T10:15:14.798478Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (30.99s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (30.99s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] 2025-06-24T10:15:15.395261Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-06-24T10:15:34.364386Z error port forward failed: lost connection to pod 2025-06-24T10:15:34.817118Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward terminated 2025-06-24T10:15:34.854764Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward reconnect success 2025-06-24T10:15:34.937153Z error port forward failed: lost connection to pod 2025-06-24T10:15:35.384818Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward terminated 2025-06-24T10:15:35.430086Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward reconnect success 2025-06-24T10:15:35.889292Z error port forward failed: lost connection to pod 2025-06-24T10:15:36.346523Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward terminated 2025-06-24T10:15:36.395306Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward reconnect success 2025-06-24T10:15:36.512549Z error port forward failed: lost connection to pod 2025-06-24T10:15:36.953452Z warn pod: external/external-v1-f9d74566f-lg7nm port forward terminated 2025-06-24T10:15:36.994770Z warn pod: external/external-v1-f9d74566f-lg7nm port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:15:45.396092Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.000825202s)' === 2025-06-24T10:15:45.396183Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:15:45.396202Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:15:45.396186Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:15:45.597985Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18b3574d15-6ae3-4ea9-8764-6675684100fc 2025-06-24T10:15:45.598244Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2017s 2025-06-24T10:15:45.598823Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:45.693501Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18bb468c7d-ed12-472f-85c8-8d0d926f1fb6 2025-06-24T10:15:45.693714Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2972s 2025-06-24T10:15:45.694323Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:45.782478Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/183fb5381c-fa39-4aab-8a80-e7e606b2689b 2025-06-24T10:15:45.782650Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3860s 2025-06-24T10:15:45.783183Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (30.90s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (30.90s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] 2025-06-24T10:15:46.295761Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-06-24T10:15:48.834040Z error port forward failed: lost connection to pod 2025-06-24T10:15:49.287530Z warn pod: echo1/proxyless-grpc-v1-8996df688-bdq7m port forward terminated 2025-06-24T10:15:49.324333Z warn pod: echo1/proxyless-grpc-v1-8996df688-bdq7m port forward reconnect success 2025-06-24T10:15:50.229775Z error port forward failed: lost connection to pod 2025-06-24T10:15:50.645613Z warn pod: echo1/client-v1-546c55dc9c-9lmdp port forward terminated 2025-06-24T10:15:50.683390Z warn pod: echo1/client-v1-546c55dc9c-9lmdp port forward reconnect success 2025-06-24T10:15:55.573122Z error port forward failed: lost connection to pod 2025-06-24T10:15:55.832555Z error port forward failed: lost connection to pod 2025-06-24T10:15:56.032471Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward terminated 2025-06-24T10:15:56.067107Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward reconnect success 2025-06-24T10:15:56.284390Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward terminated 2025-06-24T10:15:56.322522Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward reconnect success 2025-06-24T10:15:56.641544Z error port forward failed: lost connection to pod 2025-06-24T10:15:56.641591Z error port forward failed: lost connection to pod 2025-06-24T10:15:57.095443Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward terminated 2025-06-24T10:15:57.099588Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward terminated 2025-06-24T10:15:57.135645Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward reconnect success 2025-06-24T10:15:57.136318Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward reconnect success 2025-06-24T10:15:57.195777Z error port forward failed: lost connection to pod 2025-06-24T10:15:57.657207Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward terminated 2025-06-24T10:15:57.693946Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward reconnect success 2025-06-24T10:15:58.087350Z error port forward failed: lost connection to pod 2025-06-24T10:15:58.363625Z error port forward failed: lost connection to pod 2025-06-24T10:15:58.545269Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:15:58.576470Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:15:58.822189Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward terminated 2025-06-24T10:15:58.860960Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.42:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.42:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:16:16.296571Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000808771s)' === 2025-06-24T10:16:16.296676Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:16:16.296676Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:16:16.296711Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:16:16.700107Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/1893e07ce5-132c-4356-b544-bc911123bafe 2025-06-24T10:16:16.700363Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4032s 2025-06-24T10:16:16.700954Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:16.776945Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/186d86ec19-150f-45ab-891b-38b0da711b61 2025-06-24T10:16:16.777138Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4800s 2025-06-24T10:16:16.777686Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:16.862039Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/189638363d-861f-4e0f-8eee-f7f52f697505 2025-06-24T10:16:16.862157Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5650s 2025-06-24T10:16:16.862762Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.13s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.13s) === RUN TestTrustDomainValidation/From_primary-0 2025-06-24T10:14:38.791249Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:16:17.424781Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m38.633528973s)' === 2025-06-24T10:16:17.424862Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:16:17.424873Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:16:17.424915Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:16:17.595246Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/186c2ee1ba-ead3-43b9-900f-7d77589db1ef 2025-06-24T10:16:17.595516Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1705s 2025-06-24T10:16:17.596322Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:17.699698Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/182c4f0efa-f256-4052-aa4d-aca566d488a8 2025-06-24T10:16:17.699786Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2748s 2025-06-24T10:16:17.700408Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:17.771820Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/184d5964a5-df79-4c2c-98b1-a1e7bb889cac 2025-06-24T10:16:17.771955Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3469s 2025-06-24T10:16:17.772692Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (99.51s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (99.51s) === RUN TestTrustDomainValidation 2025-06-24T10:14:38.771055Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:16:18.301377Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m39.530317091s)' === 2025-06-24T10:16:18.301484Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:16:18.301502Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:16:18.301484Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:16:18.509441Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/187d3a85de-8486-48a3-9064-8c0ca4dfa018 2025-06-24T10:16:18.509708Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2080s 2025-06-24T10:16:18.510435Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:18.587206Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/1884c9bebd-8890-4c37-a67e-ba3c06275fad 2025-06-24T10:16:18.587397Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2857s 2025-06-24T10:16:18.588125Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:18.699634Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18f5961ac1-b03a-4e86-b1b4-b1e141643551 2025-06-24T10:16:18.699772Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3980s 2025-06-24T10:16:18.700547Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (100.52s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (100.52s) 2025-06-24T10:16:19.290165Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-06-24T10:16:19.290286Z info tf === Suite "security_ca_custom_root" run time: 3m4.413666355s === 2025-06-24T10:16:19.290312Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-06-24T10:16:19.291717Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:19.291727Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:19.291718Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:19.489268Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:19Z/19f0075080-54ef-4c14-9bdb-b0b4a906e9e5 2025-06-24T10:16:19.489493Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1975s 2025-06-24T10:16:19.490072Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:19.584700Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:19Z/1920b9f98c-520b-4679-99f2-74ce39bd2c11 2025-06-24T10:16:19.584880Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2928s 2025-06-24T10:16:19.585461Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:19.681798Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:19Z/19e2e29429-7e02-414e-9ee1-e296af4c61cf 2025-06-24T10:16:19.681996Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3899s 2025-06-24T10:16:19.682822Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:20.166014Z warn tf cluster/pod cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:21.741193Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:16:21.741282Z info tf === SUCCEEDED: Cleanup Istio in 83.536µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root DONE 22 tests, 9 failures in 200.481s exec: [go test -json -test.run=^TestSecureNaming$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3423377 2025-06-24T10:16:33.971085Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:16:33.971128Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:16:34.004962Z info tf === Test Framework Settings === 2025-06-24T10:16:34.005070Z info tf TestID: security_ca_custom_root RunID: 8476d9ca-7cc1-45b1-b1b4-4f98794cbe44 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:16:34.005077Z info tf =============================== 2025-06-24T10:16:34.005295Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c 2025-06-24T10:16:34.005418Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:16:34.005442Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:16:34.005475Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:16:34.005491Z info tf === BEGIN: Building clusters === 2025-06-24T10:16:34.010829Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:16:34.010880Z info tf === DONE: Building clusters === 2025-06-24T10:16:34.010900Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:16:34.044513Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:16:34.044552Z info tf === SUCCESS: context setup function === 2025-06-24T10:16:34.044559Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:16:34.044569Z info tf === Istio Component Config === 2025-06-24T10:16:34.044604Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:16:34.044610Z info tf ================================ 2025-06-24T10:16:34.047318Z info tf skipping deployment as specified in the config 2025-06-24T10:16:34.047332Z info tf === SUCCEEDED: Deploy Istio in 2.770772ms [Suite=security_ca_custom_root]=== 2025-06-24T10:16:34.047339Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:16:34.124799Z info tf === SUCCEEDED: Create namespace echo1 in 77.451789ms === Generating RSA private key, 2048 bit long modulus (2 primes) **************************************************+++++ ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c/_suite_context/ca-custom-root-2545434770/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ******************************************************************************************************************+++++ ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c/_suite_context/ca-custom-root-2545434770/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) **********+++++ **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c/_suite_context/ca-custom-root-2545434770/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:16:35.867416Z info tf === BEGIN: Create namespace external === 2025-06-24T10:16:35.912037Z info tf === SUCCEEDED: Create namespace external in 44.634998ms === 2025-06-24T10:16:36.121660Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:16:36.231324Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:16:36.231377Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c/_suite_context/ServiceAccount-Service.913873818.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-bar" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-06-24T10:16:36.231442Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:36.231438Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:36.231461Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:36.562111Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-fx9tx_echo1") 2025-06-24T10:16:36.574629Z error klog an error occurred forwarding 46331 -> 15020: error forwarding port 15020 to pod 74c732b87320d572311a94c771d82207bb2d0837eab66fee36d3a466ae1c5ac7, uid : port forward into network namespace "/var/run/netns/acdb70f5-a1b4-4771-bad5-db8e4980606e": failed to connect to localhost:15020 inside namespace 74c732b87320d572311a94c771d82207bb2d0837eab66fee36d3a466ae1c5ac7: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.577019Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s for: Get "http://localhost:46331/debug/ndsz": EOF 2025-06-24T10:16:36.577092Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.580377Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s for: Get "http://localhost:46331/stats/prometheus": dial tcp 127.0.0.1:46331: connect: connection refused 2025-06-24T10:16:36.586000Z error klog an error occurred forwarding 35841 -> 15020: error forwarding port 15020 to pod efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d, uid : port forward into network namespace "/var/run/netns/8e126e37-5d5d-40f7-a3ac-537ca041ad74": failed to connect to localhost:15020 inside namespace efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.587862Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.587886Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:35841/debug/ndsz": EOF 2025-06-24T10:16:36.588563Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:35841/stats/prometheus": dial tcp 127.0.0.1:35841: connect: connection refused 2025-06-24T10:16:36.599750Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-j7j56_echo1") 2025-06-24T10:16:36.600435Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-4gdpk_echo1") 2025-06-24T10:16:36.601446Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6887fb7d79-2xfrl_echo1") 2025-06-24T10:16:36.629845Z error klog an error occurred forwarding 40655 -> 15020: error forwarding port 15020 to pod 6d1cbbb559dfc1f98882e584d1eada273fe5da4c861fd619e891968840aa025a, uid : port forward into network namespace "/var/run/netns/27062226-6025-41f5-8eb3-9391bf3b294b": failed to connect to localhost:15020 inside namespace 6d1cbbb559dfc1f98882e584d1eada273fe5da4c861fd619e891968840aa025a: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.629935Z error klog an error occurred forwarding 33251 -> 15020: error forwarding port 15020 to pod 0a1753cb73de6dab13ed0d5b3f8738f4dd6858107eab53331373b561d53f11c2, uid : port forward into network namespace "/var/run/netns/9771b388-c628-4757-aae5-638ba2a34421": failed to connect to localhost:15020 inside namespace 0a1753cb73de6dab13ed0d5b3f8738f4dd6858107eab53331373b561d53f11c2: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.629981Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:36Z/2493eedfbe-b035-4fd0-aa4a-416ed2b7bf04 2025-06-24T10:16:36.630170Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3983s 2025-06-24T10:16:36.630767Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:36.631360Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.631380Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-crwrb for: Get "http://localhost:40655/debug/ndsz": EOF 2025-06-24T10:16:36.631436Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.631470Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-d8h2w for: Get "http://localhost:33251/debug/ndsz": EOF 2025-06-24T10:16:36.688296Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-fx9tx_echo1") 2025-06-24T10:16:36.721787Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-4gdpk_echo1") 2025-06-24T10:16:36.721856Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6887fb7d79-2xfrl_echo1") 2025-06-24T10:16:36.722748Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-d8h2w for: Get "http://localhost:33251/stats/prometheus": dial tcp 127.0.0.1:33251: connect: connection refused 2025-06-24T10:16:36.725316Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-crwrb for: Get "http://localhost:40655/stats/prometheus": dial tcp 127.0.0.1:40655: connect: connection refused 2025-06-24T10:16:36.727210Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-j7j56_echo1") 2025-06-24T10:16:36.758739Z error klog an error occurred forwarding 39201 -> 15020: error forwarding port 15020 to pod c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a, uid : port forward into network namespace "/var/run/netns/f50c2518-399b-446e-a479-92efcf07626c": failed to connect to localhost:15020 inside namespace c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.760720Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.761228Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:39201/debug/ndsz": EOF 2025-06-24T10:16:36.761734Z error klog an error occurred forwarding 40411 -> 15020: error forwarding port 15020 to pod b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008, uid : port forward into network namespace "/var/run/netns/19206d28-bacc-4283-91da-7699826c080c": failed to connect to localhost:15020 inside namespace b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.761749Z error klog an error occurred forwarding 42143 -> 15020: error forwarding port 15020 to pod c0e0350d456b568df9651e032518cdc778d06fe735862b243eba001aad5221d6, uid : port forward into network namespace "/var/run/netns/5c043942-5ad8-4ec0-9958-e2e7a0ec3ef1": failed to connect to localhost:15020 inside namespace c0e0350d456b568df9651e032518cdc778d06fe735862b243eba001aad5221d6: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.761969Z error klog an error occurred forwarding 35407 -> 15020: error forwarding port 15020 to pod e7f94b0a6f75982e49a9c085fd1f2123de626f42c165eee5f683f99d234b054b, uid : port forward into network namespace "/var/run/netns/d463e655-924b-4927-a6d5-fdd6f933c5c8": failed to connect to localhost:15020 inside namespace e7f94b0a6f75982e49a9c085fd1f2123de626f42c165eee5f683f99d234b054b: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:36.761996Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:39201/stats/prometheus": dial tcp 127.0.0.1:39201: connect: connection refused 2025-06-24T10:16:36.763308Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:42143/debug/ndsz": EOF 2025-06-24T10:16:36.763359Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.763419Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.763435Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:40411/debug/ndsz": EOF 2025-06-24T10:16:36.763543Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-q5jkr for: Get "http://localhost:35407/debug/ndsz": EOF 2025-06-24T10:16:36.763581Z error port forward failed: lost connection to pod 2025-06-24T10:16:36.763891Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-q5jkr for: Get "http://localhost:35407/stats/prometheus": dial tcp 127.0.0.1:35407: connect: connection refused 2025-06-24T10:16:36.763930Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:42143/stats/prometheus": dial tcp 127.0.0.1:42143: connect: connection refused 2025-06-24T10:16:36.764035Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:40411/stats/prometheus": dial tcp 127.0.0.1:40411: connect: connection refused 2025-06-24T10:16:36.822211Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-fx9tx_echo1") 2025-06-24T10:16:36.824983Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-j7j56_echo1") 2025-06-24T10:16:36.827703Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6887fb7d79-2xfrl_echo1") 2025-06-24T10:16:36.831721Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-4gdpk_echo1") 2025-06-24T10:16:36.882310Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:36Z/24e67c314d-6955-4a2d-9a21-7ca3bdebeb0f 2025-06-24T10:16:36.882440Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6505s 2025-06-24T10:16:36.882967Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:36.909267Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-fx9tx_echo1") 2025-06-24T10:16:36.909507Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-j7j56_echo1") 2025-06-24T10:16:36.909803Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6887fb7d79-2xfrl_echo1") 2025-06-24T10:16:36.915717Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-4gdpk_echo1") 2025-06-24T10:16:36.978789Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-fx9tx_echo1") 2025-06-24T10:16:36.982152Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-4gdpk_echo1") 2025-06-24T10:16:36.982743Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-j7j56_echo1") 2025-06-24T10:16:36.982895Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6887fb7d79-2xfrl_echo1") 2025-06-24T10:16:36.993853Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:16:37.037881Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:36Z/24af32281b-0519-4ac8-a6c0-4fd17a1095d3 2025-06-24T10:16:37.037967Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8061s 2025-06-24T10:16:37.038672Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:37.123304Z warn tf cluster/pod cluster-0/istio-system/istio-ingressgateway-bf7cd59d6-64wm6 found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:37.270081Z warn tf cluster/pod cluster-0/istio-system/istio-egressgateway-5454f7b8bd-24g9b found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:37.394608Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-8996df688-bdq7m/istio-proxy for: pods "proxyless-grpc-v1-8996df688-bdq7m" not found 2025-06-24T10:16:37.399112Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-8996df688-bdq7m/app for: pods "proxyless-grpc-v1-8996df688-bdq7m" not found 2025-06-24T10:16:37.543106Z warn tf cluster/pod cluster-0/echo1/a-v1-78f49b749b-crwrb found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:37.627102Z warn tf cluster/pod cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:37.842568Z warn tf cluster/pod cluster-0/echo1/server-v1-86cfb5b889-rlhjn found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:37.853431Z warn tf cluster/pod cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:37.888323Z warn tf cluster/pod cluster-0/echo1/client-v1-546c55dc9c-9lmdp found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:37.968554Z warn tf cluster/pod cluster-0/echo1/b-v1-7bcb76fc6f-d8h2w found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:16:38.014680Z error klog an error occurred forwarding 43681 -> 15000: error forwarding port 15000 to pod c0e0350d456b568df9651e032518cdc778d06fe735862b243eba001aad5221d6, uid : port forward into network namespace "/var/run/netns/5c043942-5ad8-4ec0-9958-e2e7a0ec3ef1": writeto tcp 127.0.0.1:41524->127.0.0.1:15000: read tcp 127.0.0.1:41524->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:16:38.016896Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43681/config_dump?include_eds=true": EOF 2025-06-24T10:16:38.016985Z error port forward failed: lost connection to pod 2025-06-24T10:16:38.017539Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43681/clusters": dial tcp 127.0.0.1:43681: connect: connection refused 2025-06-24T10:16:38.549812Z error klog an error occurred forwarding 43049 -> 15000: error forwarding port 15000 to pod 6d1cbbb559dfc1f98882e584d1eada273fe5da4c861fd619e891968840aa025a, uid : port forward into network namespace "/var/run/netns/27062226-6025-41f5-8eb3-9391bf3b294b": failed to connect to localhost:15000 inside namespace 6d1cbbb559dfc1f98882e584d1eada273fe5da4c861fd619e891968840aa025a: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:38.551820Z error port forward failed: lost connection to pod 2025-06-24T10:16:38.551828Z error tf FAIL: Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-crwrb for: Get "http://localhost:43049/config_dump?include_eds=true": EOF 2025-06-24T10:16:38.552719Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-crwrb for: Get "http://localhost:43049/clusters": dial tcp 127.0.0.1:43049: connect: connection refused 2025-06-24T10:16:38.633489Z error klog an error occurred forwarding 36091 -> 15000: error forwarding port 15000 to pod efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d, uid : port forward into network namespace "/var/run/netns/8e126e37-5d5d-40f7-a3ac-537ca041ad74": failed to connect to localhost:15000 inside namespace efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:38.635677Z error tf FAIL: Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:36091/config_dump?include_eds=true": EOF 2025-06-24T10:16:38.635751Z error port forward failed: lost connection to pod 2025-06-24T10:16:38.636619Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:36091/clusters": dial tcp 127.0.0.1:36091: connect: connection refused 2025-06-24T10:16:38.850087Z error klog an error occurred forwarding 42597 -> 15000: error forwarding port 15000 to pod c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a, uid : port forward into network namespace "/var/run/netns/f50c2518-399b-446e-a479-92efcf07626c": failed to connect to localhost:15000 inside namespace c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:38.852426Z error tf FAIL: Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:42597/config_dump?include_eds=true": EOF 2025-06-24T10:16:38.852489Z error port forward failed: lost connection to pod 2025-06-24T10:16:38.853138Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:42597/clusters": dial tcp 127.0.0.1:42597: connect: connection refused 2025-06-24T10:16:38.861131Z error klog an error occurred forwarding 43267 -> 15000: error forwarding port 15000 to pod 74c732b87320d572311a94c771d82207bb2d0837eab66fee36d3a466ae1c5ac7, uid : port forward into network namespace "/var/run/netns/acdb70f5-a1b4-4771-bad5-db8e4980606e": failed to connect to localhost:15000 inside namespace 74c732b87320d572311a94c771d82207bb2d0837eab66fee36d3a466ae1c5ac7: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:38.863091Z error tf FAIL: Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s for: Get "http://localhost:43267/config_dump?include_eds=true": EOF 2025-06-24T10:16:38.863107Z error port forward failed: lost connection to pod 2025-06-24T10:16:38.863877Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s for: Get "http://localhost:43267/clusters": dial tcp 127.0.0.1:43267: connect: connection refused 2025-06-24T10:16:38.896513Z error klog an error occurred forwarding 44785 -> 15000: error forwarding port 15000 to pod b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008, uid : port forward into network namespace "/var/run/netns/19206d28-bacc-4283-91da-7699826c080c": failed to connect to localhost:15000 inside namespace b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:38.898755Z error port forward failed: lost connection to pod 2025-06-24T10:16:38.898757Z error tf FAIL: Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:44785/config_dump?include_eds=true": EOF 2025-06-24T10:16:38.899562Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:44785/clusters": dial tcp 127.0.0.1:44785: connect: connection refused 2025-06-24T10:16:38.981147Z error klog an error occurred forwarding 34291 -> 15000: error forwarding port 15000 to pod 0a1753cb73de6dab13ed0d5b3f8738f4dd6858107eab53331373b561d53f11c2, uid : port forward into network namespace "/var/run/netns/9771b388-c628-4757-aae5-638ba2a34421": failed to connect to localhost:15000 inside namespace 0a1753cb73de6dab13ed0d5b3f8738f4dd6858107eab53331373b561d53f11c2: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:38.983155Z error tf FAIL: Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-d8h2w for: Get "http://localhost:34291/config_dump?include_eds=true": EOF 2025-06-24T10:16:38.983253Z error port forward failed: lost connection to pod 2025-06-24T10:16:38.983913Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-d8h2w for: Get "http://localhost:34291/clusters": dial tcp 127.0.0.1:34291: connect: connection refused 2025-06-24T10:16:39.011645Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c/_suite_context/ServiceAccount-Service.913873818.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-bar" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-06-24T10:16:39.011669Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c/_suite_context/ServiceAccount-Service.913873818.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-bar" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-06-24T10:16:39.011675Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8476d9c/_suite_context/ServiceAccount-Service.913873818.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-bar" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-06-24T10:16:39.011717Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:39.011724Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:39.011705Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:16:39.184109Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-06-24T10:16:39.194410Z error klog an error occurred forwarding 39991 -> 15000: error forwarding port 15000 to pod 0a1753cb73de6dab13ed0d5b3f8738f4dd6858107eab53331373b561d53f11c2, uid : sandbox 0a1753cb73de6dab13ed0d5b3f8738f4dd6858107eab53331373b561d53f11c2 is not running: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.195991Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-d8h2w for: Get "http://localhost:39991/config_dump?include_eds=true": EOF 2025-06-24T10:16:39.196027Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.196804Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-d8h2w for: Get "http://localhost:39991/clusters": dial tcp 127.0.0.1:39991: connect: connection refused 2025-06-24T10:16:39.197556Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-06-24T10:16:39.200909Z error klog an error occurred forwarding 37821 -> 15020: error forwarding port 15020 to pod b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008, uid : port forward into network namespace "/var/run/netns/19206d28-bacc-4283-91da-7699826c080c": failed to connect to localhost:15020 inside namespace b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.203159Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.203371Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:37821/debug/ndsz": EOF 2025-06-24T10:16:39.204639Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:37821/stats/prometheus": dial tcp 127.0.0.1:37821: connect: connection refused 2025-06-24T10:16:39.215786Z error klog an error occurred forwarding 41769 -> 15000: error forwarding port 15000 to pod b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008, uid : port forward into network namespace "/var/run/netns/19206d28-bacc-4283-91da-7699826c080c": failed to connect to localhost:15000 inside namespace b38866e2a5519f2de309ab7b59912b0f394d691bcb0ab28c4214fc041aedf008: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.215976Z error klog an error occurred forwarding 43575 -> 15020: error forwarding port 15020 to pod c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a, uid : port forward into network namespace "/var/run/netns/f50c2518-399b-446e-a479-92efcf07626c": failed to connect to localhost:15020 inside namespace c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.217946Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:43575/debug/ndsz": EOF 2025-06-24T10:16:39.217947Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.217981Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:41769/config_dump?include_eds=true": EOF 2025-06-24T10:16:39.218116Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.218693Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-9lmdp for: Get "http://localhost:41769/clusters": dial tcp 127.0.0.1:41769: connect: connection refused 2025-06-24T10:16:39.218772Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:43575/stats/prometheus": dial tcp 127.0.0.1:43575: connect: connection refused 2025-06-24T10:16:39.223659Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-64d65d6c8b-wdr9s_echo1") 2025-06-24T10:16:39.228952Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-crwrb_echo1") 2025-06-24T10:16:39.230867Z error klog an error occurred forwarding 35577 -> 15000: error forwarding port 15000 to pod e7f94b0a6f75982e49a9c085fd1f2123de626f42c165eee5f683f99d234b054b, uid : port forward into network namespace "/var/run/netns/d463e655-924b-4927-a6d5-fdd6f933c5c8": failed to connect to localhost:15000 inside namespace e7f94b0a6f75982e49a9c085fd1f2123de626f42c165eee5f683f99d234b054b: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.231710Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-64d65d6c8b-wdr9s_echo1") 2025-06-24T10:16:39.233596Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-q5jkr for: Get "http://localhost:35577/config_dump?include_eds=true": EOF 2025-06-24T10:16:39.233629Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.234348Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-q5jkr for: Get "http://localhost:35577/clusters": dial tcp 127.0.0.1:35577: connect: connection refused 2025-06-24T10:16:39.243882Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-7bcb76fc6f-d8h2w_echo1") 2025-06-24T10:16:39.244556Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-crwrb_echo1") 2025-06-24T10:16:39.253926Z error klog an error occurred forwarding 45557 -> 15020: error forwarding port 15020 to pod efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d, uid : port forward into network namespace "/var/run/netns/8e126e37-5d5d-40f7-a3ac-537ca041ad74": failed to connect to localhost:15020 inside namespace efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.254424Z error klog an error occurred forwarding 34957 -> 15000: error forwarding port 15000 to pod c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a, uid : port forward into network namespace "/var/run/netns/f50c2518-399b-446e-a479-92efcf07626c": failed to connect to localhost:15000 inside namespace c1ae024e2a31baa115a695a5659029f4e177f8ed6399b174204959503f0b8a4a: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.255784Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:45557/debug/ndsz": EOF 2025-06-24T10:16:39.255893Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.255970Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:34957/config_dump?include_eds=true": EOF 2025-06-24T10:16:39.256031Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.256357Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:45557/stats/prometheus": dial tcp 127.0.0.1:45557: connect: connection refused 2025-06-24T10:16:39.256410Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-rlhjn for: Get "http://localhost:34957/clusters": dial tcp 127.0.0.1:34957: connect: connection refused 2025-06-24T10:16:39.258046Z error klog an error occurred forwarding 37059 -> 15000: error forwarding port 15000 to pod efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d, uid : port forward into network namespace "/var/run/netns/8e126e37-5d5d-40f7-a3ac-537ca041ad74": failed to connect to localhost:15000 inside namespace efa82a562e894b0fbc8941d9c0b8c0f94e974994b7af149866e87fdbae3b131d: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.259729Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:37059/config_dump?include_eds=true": EOF 2025-06-24T10:16:39.259830Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.260233Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc for: Get "http://localhost:37059/clusters": dial tcp 127.0.0.1:37059: connect: connection refused 2025-06-24T10:16:39.275462Z error klog an error occurred forwarding 33899 -> 15020: error forwarding port 15020 to pod e7f94b0a6f75982e49a9c085fd1f2123de626f42c165eee5f683f99d234b054b, uid : port forward into network namespace "/var/run/netns/d463e655-924b-4927-a6d5-fdd6f933c5c8": failed to connect to localhost:15020 inside namespace e7f94b0a6f75982e49a9c085fd1f2123de626f42c165eee5f683f99d234b054b: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:16:39.276045Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-06-24T10:16:39.277528Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-q5jkr for: Get "http://localhost:33899/debug/ndsz": EOF 2025-06-24T10:16:39.277581Z error port forward failed: lost connection to pod 2025-06-24T10:16:39.278392Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-q5jkr for: Get "http://localhost:33899/stats/prometheus": dial tcp 127.0.0.1:33899: connect: connection refused 2025-06-24T10:16:39.292494Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-64d65d6c8b-wdr9s_echo1") 2025-06-24T10:16:39.311827Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-06-24T10:16:39.318309Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:36Z/24974bb029-d0fe-41d9-a0c4-c90b74119155 2025-06-24T10:16:39.318503Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3065s 2025-06-24T10:16:39.319459Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:39.321181Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-crwrb_echo1") 2025-06-24T10:16:39.322082Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-64d65d6c8b-wdr9s_echo1") 2025-06-24T10:16:39.323614Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-7bcb76fc6f-d8h2w_echo1") 2025-06-24T10:16:39.348188Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-crwrb_echo1") 2025-06-24T10:16:39.361086Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-06-24T10:16:39.364102Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-64d65d6c8b-wdr9s_echo1") 2025-06-24T10:16:39.424779Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-7bcb76fc6f-d8h2w_echo1") 2025-06-24T10:16:39.428276Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-crwrb_echo1") 2025-06-24T10:16:39.433117Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-06-24T10:16:39.472409Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-crwrb_echo1") 2025-06-24T10:16:39.502057Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-crwrb_echo1") 2025-06-24T10:16:39.509555Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:36Z/24af0fb71a-5f52-4d62-836c-910953ed4e0c 2025-06-24T10:16:39.509574Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-06-24T10:16:39.509675Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4976s 2025-06-24T10:16:39.510369Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:39.545527Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-64d65d6c8b-wdr9s" not found in the "echo1" namespace) 2025-06-24T10:16:39.545520Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-64d65d6c8b-wdr9s" not found in the "echo1" namespace) 2025-06-24T10:16:39.642355Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:16:36Z/24d72a5e00-3009-4690-b881-56c276a4a6ce 2025-06-24T10:16:39.642509Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6305s 2025-06-24T10:16:39.643187Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:39.652830Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s/app for: pods "headless-v1-64d65d6c8b-wdr9s" not found 2025-06-24T10:16:39.668692Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s/istio-proxy for: pods "headless-v1-64d65d6c8b-wdr9s" not found 2025-06-24T10:16:39.673850Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-64d65d6c8b-wdr9s/istio-validation for: pods "headless-v1-64d65d6c8b-wdr9s" not found 2025-06-24T10:16:39.901122Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc/app for: pods "sotw-v1-65f898ccd5-mh5qc" not found 2025-06-24T10:16:39.910475Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc/istio-proxy for: pods "sotw-v1-65f898ccd5-mh5qc" not found 2025-06-24T10:16:39.913787Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-mh5qc/istio-validation for: pods "sotw-v1-65f898ccd5-mh5qc" not found 2025-06-24T10:16:39.950650Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/app for: the server could not find the requested resource ( pods/log statefulset-v1-0) 2025-06-24T10:16:39.955808Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-proxy for: pods "statefulset-v1-0" not found 2025-06-24T10:16:39.959477Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-validation for: pods "statefulset-v1-0" not found 2025-06-24T10:16:39.994043Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:16:39.994106Z info tf === SUCCEEDED: Cleanup Istio in 51.572µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0/server:server-naked-foo\[success\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3424723 2025-06-24T10:16:50.845787Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:16:50.845869Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:16:50.882107Z info tf === Test Framework Settings === 2025-06-24T10:16:50.882210Z info tf TestID: security_ca_custom_root RunID: da4c171b-d36f-4a53-9307-dd231528c4ac NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:16:50.882216Z info tf =============================== 2025-06-24T10:16:50.882357Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-da4c171 2025-06-24T10:16:50.882561Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:16:50.882577Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:16:50.882592Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:16:50.882600Z info tf === BEGIN: Building clusters === 2025-06-24T10:16:50.885114Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:16:50.885123Z info tf === DONE: Building clusters === 2025-06-24T10:16:50.885142Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:16:50.921771Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:16:50.921900Z info tf === SUCCESS: context setup function === 2025-06-24T10:16:50.921912Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:16:50.921937Z info tf === Istio Component Config === 2025-06-24T10:16:50.921997Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:16:50.922005Z info tf ================================ 2025-06-24T10:16:50.928334Z info tf skipping deployment as specified in the config 2025-06-24T10:16:50.928378Z info tf === SUCCEEDED: Deploy Istio in 6.457557ms [Suite=security_ca_custom_root]=== 2025-06-24T10:16:50.928394Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:16:51.010799Z info tf === SUCCEEDED: Create namespace echo1 in 82.350921ms === Generating RSA private key, 2048 bit long modulus (2 primes) ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-da4c171/_suite_context/ca-custom-root-3764287388/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-da4c171/_suite_context/ca-custom-root-3764287388/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *************************************************************************************************************************************************************************************************************************************************************************************+++++ ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-da4c171/_suite_context/ca-custom-root-3764287388/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:16:52.776173Z info tf === BEGIN: Create namespace external === 2025-06-24T10:16:52.800930Z info tf === SUCCEEDED: Create namespace external in 24.777735ms === 2025-06-24T10:16:52.977929Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:16:54.834275Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:54.935519Z info klog Caches are synced for echo 2025-06-24T10:16:54.943049Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:54.948995Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:54.983135Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.008985Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.043660Z info klog Caches are synced for echo 2025-06-24T10:16:55.049212Z info klog Caches are synced for echo 2025-06-24T10:16:55.051618Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.055890Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.063840Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.083397Z info klog Caches are synced for echo 2025-06-24T10:16:55.109500Z info klog Caches are synced for echo 2025-06-24T10:16:55.117082Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.128003Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.152468Z info klog Caches are synced for echo 2025-06-24T10:16:55.156179Z info klog Caches are synced for echo 2025-06-24T10:16:55.164863Z info klog Caches are synced for echo 2025-06-24T10:16:55.174827Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.217340Z info klog Caches are synced for echo 2025-06-24T10:16:55.228804Z info klog Caches are synced for echo 2025-06-24T10:16:55.252100Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.275532Z info klog Caches are synced for echo 2025-06-24T10:16:55.291039Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.292973Z info klog Waiting for caches to sync for echo 2025-06-24T10:16:55.352903Z info klog Caches are synced for echo 2025-06-24T10:16:55.391330Z info klog Caches are synced for echo 2025-06-24T10:16:55.394045Z info klog Caches are synced for echo 2025-06-24T10:17:23.866324Z info tf === SUCCEEDED: Deploy echo instances in 30.888380394s === 2025-06-24T10:17:23.900069Z info tf === DONE: Setup: 'security_ca_custom_root' (33.01491123s) === 2025-06-24T10:17:23.900108Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] 2025-06-24T10:17:23.918025Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.21.3.128:47180->172.30.198.181:443: read: connection reset by peer 2025-06-24T10:17:53.924448Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006418727s)' === 2025-06-24T10:17:53.924513Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:17:53.924520Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:17:53.924550Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:17:54.252196Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/354a17000a-f8c1-40e5-a03c-12406f2cf890 2025-06-24T10:17:54.252470Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3277s 2025-06-24T10:17:54.254035Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:54.379799Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35cf8c0d85-1883-424c-8aba-66260fa8dc7b 2025-06-24T10:17:54.379941Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4552s 2025-06-24T10:17:54.380781Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:54.472243Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/359c2940d9-afa3-4492-9b36-3f69f7dba078 2025-06-24T10:17:54.472441Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5476s 2025-06-24T10:17:54.472940Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (31.17s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 1) (31.17s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0 2025-06-24T10:17:23.917793Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-06-24T10:17:55.085042Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (31.167241292s)' === 2025-06-24T10:17:55.085152Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:17:55.085154Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:17:55.085171Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:17:55.336675Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35238225c5-4d29-4686-abcb-3f1790e68b3e 2025-06-24T10:17:55.336961Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2516s 2025-06-24T10:17:55.337428Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:55.446369Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/359bc4c8ea-8c2e-4c1f-864d-8c92e54a3415 2025-06-24T10:17:55.446548Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3612s 2025-06-24T10:17:55.447262Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:55.530884Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35620451f7-c105-4875-9f0d-bd29d22731ef 2025-06-24T10:17:55.531103Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4458s 2025-06-24T10:17:55.531730Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0 (32.41s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 1) (32.41s) === RUN TestTrustDomainAliasSecureNaming 2025-06-24T10:17:23.900535Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-06-24T10:17:56.324206Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.423663615s)' === 2025-06-24T10:17:56.324335Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:17:56.324349Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:17:56.324367Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:17:56.522168Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/355700bc02-4828-4ebb-b45e-d398927b9189 2025-06-24T10:17:56.522483Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1979s 2025-06-24T10:17:56.522801Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:56.615289Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/350c87d4e4-019d-4cf5-a9b0-4e73dddeaa92 2025-06-24T10:17:56.615467Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2909s 2025-06-24T10:17:56.616035Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:56.744268Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35182b1fe9-034f-45ae-8088-f557640f7912 2025-06-24T10:17:56.744452Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4199s 2025-06-24T10:17:56.745137Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming (33.80s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming (re-run 1) (33.80s) 2025-06-24T10:17:57.697785Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-06-24T10:17:57.697857Z info tf === Suite "security_ca_custom_root" run time: 1m6.812718615s === 2025-06-24T10:17:57.697874Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-06-24T10:17:57.698243Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:17:57.698246Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:17:57.698245Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:17:57.912683Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:57Z/369cdeda0e-2cbe-450a-8096-502ba660b2e6 2025-06-24T10:17:57.913000Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2144s 2025-06-24T10:17:57.913649Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:58.039922Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:57Z/361457eb01-fcb1-4fba-97af-9d4179c0a2e8 2025-06-24T10:17:58.040142Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3415s 2025-06-24T10:17:58.040927Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:58.156157Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:57Z/36fe3a9736-21f5-4cec-be1c-7e7a434d9d8c 2025-06-24T10:17:58.156406Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4578s 2025-06-24T10:17:58.157077Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:18:04.731958Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:18:04.732044Z info tf === SUCCEEDED: Cleanup Istio in 76.266µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3426167 2025-06-24T10:18:16.018506Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:18:16.018555Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:18:16.051492Z info tf === Test Framework Settings === 2025-06-24T10:18:16.052344Z info tf TestID: security_ca_custom_root RunID: 84d93dad-1e3a-4ee3-bc90-b3ac2bdc5481 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:18:16.052855Z info tf =============================== 2025-06-24T10:18:16.053762Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da 2025-06-24T10:18:16.053878Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:18:16.053894Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:18:16.053914Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:18:16.053925Z info tf === BEGIN: Building clusters === 2025-06-24T10:18:16.057888Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:18:16.057902Z info tf === DONE: Building clusters === 2025-06-24T10:18:16.057933Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:18:16.241447Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:18:16.241476Z info tf === SUCCESS: context setup function === 2025-06-24T10:18:16.241485Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:18:16.241515Z info tf === Istio Component Config === 2025-06-24T10:18:16.241565Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:18:16.241571Z info tf ================================ 2025-06-24T10:18:16.246704Z info tf skipping deployment as specified in the config 2025-06-24T10:18:16.246730Z info tf === SUCCEEDED: Deploy Istio in 5.241711ms [Suite=security_ca_custom_root]=== 2025-06-24T10:18:16.246739Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:18:16.285482Z info tf === SUCCEEDED: Create namespace echo1 in 38.734606ms === Generating RSA private key, 2048 bit long modulus (2 primes) ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *****************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da/_suite_context/ca-custom-root-4220883776/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *****************************************************************************************************************************************************************************************************************************************************************+++++ *****************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da/_suite_context/ca-custom-root-4220883776/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ********************+++++ ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da/_suite_context/ca-custom-root-4220883776/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:18:18.266882Z info tf === BEGIN: Create namespace external === 2025-06-24T10:18:18.278239Z info tf === SUCCEEDED: Create namespace external in 11.365817ms === 2025-06-24T10:18:18.354419Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:18:18.374616Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:18:18.374686Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da/_suite_context/Service.3099076524.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:18:18.374723Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:18:18.374763Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:18:18.374777Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:18:18.508525Z error klog an error occurred forwarding 39249 -> 15020: error forwarding port 15020 to pod be8407a51952c31a6af33786e4e4eb4bda30dfc8a4f84c2125d58929d8b44eff, uid : port forward into network namespace "/var/run/netns/d5ed3f10-3f79-47cb-81fb-ad0bfe80749e": failed to connect to localhost:15020 inside namespace be8407a51952c31a6af33786e4e4eb4bda30dfc8a4f84c2125d58929d8b44eff: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.510892Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.510883Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-shxct for: Get "http://localhost:39249/debug/ndsz": EOF 2025-06-24T10:18:18.511633Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-shxct for: Get "http://localhost:39249/stats/prometheus": dial tcp 127.0.0.1:39249: connect: connection refused 2025-06-24T10:18:18.596065Z error klog an error occurred forwarding 37449 -> 15020: error forwarding port 15020 to pod efd760a22833edbfc002d96494f113f44e96070b7a9cd49c25b54c0cf2e28678, uid : port forward into network namespace "/var/run/netns/a28e4031-b76c-418b-8cb0-c9b148a63883": failed to connect to localhost:15020 inside namespace efd760a22833edbfc002d96494f113f44e96070b7a9cd49c25b54c0cf2e28678: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.598470Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-x7bdb for: Get "http://localhost:37449/debug/ndsz": EOF 2025-06-24T10:18:18.598529Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.599477Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-x7bdb for: Get "http://localhost:37449/stats/prometheus": dial tcp 127.0.0.1:37449: connect: connection refused 2025-06-24T10:18:18.603164Z error klog an error occurred forwarding 45695 -> 15020: error forwarding port 15020 to pod 9219b26d84313691cb68ed01fac113d6565c42ac4b279754d085923d07a9965a, uid : port forward into network namespace "/var/run/netns/c8fe46a1-a7e5-4518-87d1-907b62af40a8": failed to connect to localhost:15020 inside namespace 9219b26d84313691cb68ed01fac113d6565c42ac4b279754d085923d07a9965a: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.605781Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.605807Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-dxk6g for: Get "http://localhost:45695/debug/ndsz": EOF 2025-06-24T10:18:18.606909Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-dxk6g for: Get "http://localhost:45695/stats/prometheus": dial tcp 127.0.0.1:45695: connect: connection refused 2025-06-24T10:18:18.606933Z error klog an error occurred forwarding 36025 -> 15020: error forwarding port 15020 to pod c56b51e6328d16fd3656d35122a4dd05bf71dd9d85a845c55d730c981291e6f1, uid : port forward into network namespace "/var/run/netns/755e18eb-85d2-4f4a-88d3-e6d3deb75d4d": failed to connect to localhost:15020 inside namespace c56b51e6328d16fd3656d35122a4dd05bf71dd9d85a845c55d730c981291e6f1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.607097Z error klog an error occurred forwarding 43887 -> 15020: error forwarding port 15020 to pod 881dbfc1b37538d3232f5ebddf62c2ea243384a4e8db1fb66c616076a6c90cef, uid : port forward into network namespace "/var/run/netns/51dcca1f-4bf1-46d7-928b-e878fb5d40f4": failed to connect to localhost:15020 inside namespace 881dbfc1b37538d3232f5ebddf62c2ea243384a4e8db1fb66c616076a6c90cef: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.608583Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.608676Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:36025/debug/ndsz": EOF 2025-06-24T10:18:18.608824Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-gspsg for: Get "http://localhost:43887/debug/ndsz": EOF 2025-06-24T10:18:18.608838Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.609892Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-gspsg for: Get "http://localhost:43887/stats/prometheus": dial tcp 127.0.0.1:43887: connect: connection refused 2025-06-24T10:18:18.609894Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:36025/stats/prometheus": dial tcp 127.0.0.1:36025: connect: connection refused 2025-06-24T10:18:18.611362Z error klog an error occurred forwarding 38869 -> 15020: error forwarding port 15020 to pod 6a4558d83cbe81139b54ccba61e3cf2f68e06185192bf831070072bc2b35bb21, uid : port forward into network namespace "/var/run/netns/0eb01ea0-e2f6-42a5-b186-a423fdf75f60": failed to connect to localhost:15020 inside namespace 6a4558d83cbe81139b54ccba61e3cf2f68e06185192bf831070072bc2b35bb21: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.613295Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-4q4sz for: Get "http://localhost:38869/debug/ndsz": EOF 2025-06-24T10:18:18.613379Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.614457Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-4q4sz for: Get "http://localhost:38869/stats/prometheus": dial tcp 127.0.0.1:38869: connect: connection refused 2025-06-24T10:18:18.618910Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:15Z/40ce8a090e-eb95-4605-91b4-3ef14d0d8ce2 2025-06-24T10:18:18.619131Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2440s 2025-06-24T10:18:18.619861Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:18:18.634614Z error klog an error occurred forwarding 42649 -> 15020: error forwarding port 15020 to pod 7f047329b2410823cfbf4cc5f22ee42cb73448dfaf3ffd46ae30d3774dfdaf5b, uid : port forward into network namespace "/var/run/netns/c6fd4d93-be43-40e4-9f16-9ba150480af2": failed to connect to localhost:15020 inside namespace 7f047329b2410823cfbf4cc5f22ee42cb73448dfaf3ffd46ae30d3774dfdaf5b: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.634719Z error klog an error occurred forwarding 42767 -> 15020: error forwarding port 15020 to pod 3b2eea046cdb1f916f344f96bbb8d6961b5647aef9d0b4afd229e0755feb6cfc, uid : port forward into network namespace "/var/run/netns/4744c60a-6384-41b5-8dcc-d20a11779018": failed to connect to localhost:15020 inside namespace 3b2eea046cdb1f916f344f96bbb8d6961b5647aef9d0b4afd229e0755feb6cfc: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.635054Z error klog an error occurred forwarding 42125 -> 15020: error forwarding port 15020 to pod 0eeecc4db93da01c98095778cd209b44e048be842a9c035e30e105b33facaf56, uid : port forward into network namespace "/var/run/netns/53bd180f-4765-4097-b00b-7fd9cdad9ea5": failed to connect to localhost:15020 inside namespace 0eeecc4db93da01c98095778cd209b44e048be842a9c035e30e105b33facaf56: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:18.636567Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-9btd8 for: Get "http://localhost:42767/debug/ndsz": EOF 2025-06-24T10:18:18.636604Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.636744Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-9nzhq for: Get "http://localhost:42649/debug/ndsz": EOF 2025-06-24T10:18:18.636720Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.636858Z error port forward failed: lost connection to pod 2025-06-24T10:18:18.636867Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/proxyless-grpc-v1-8996df688-kbsqm for: Get "http://localhost:42125/debug/ndsz": EOF 2025-06-24T10:18:18.637523Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-9nzhq for: Get "http://localhost:42649/stats/prometheus": dial tcp 127.0.0.1:42649: connect: connection refused 2025-06-24T10:18:18.637613Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/proxyless-grpc-v1-8996df688-kbsqm for: Get "http://localhost:42125/stats/prometheus": dial tcp 127.0.0.1:42125: connect: connection refused 2025-06-24T10:18:18.637612Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-9btd8 for: Get "http://localhost:42767/stats/prometheus": dial tcp 127.0.0.1:42767: connect: connection refused 2025-06-24T10:18:18.711236Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:15Z/40136d2219-e230-414b-a9b2-c0514359c1b2 2025-06-24T10:18:18.711325Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3362s 2025-06-24T10:18:18.712067Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:18:18.800021Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:15Z/40d5b0eca7-e7a5-4a37-acbe-1c418f0508be 2025-06-24T10:18:18.800168Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4250s 2025-06-24T10:18:18.800741Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:18:19.503485Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da/_suite_context/Service.3099076524.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:18:19.503540Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da/_suite_context/Service.3099076524.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-06-24T10:18:19.503547Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-84d93da/_suite_context/Service.3099076524.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:18:19.503572Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:18:19.503581Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:18:19.503629Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:18:19.643192Z error klog an error occurred forwarding 36323 -> 15020: error forwarding port 15020 to pod c56b51e6328d16fd3656d35122a4dd05bf71dd9d85a845c55d730c981291e6f1, uid : port forward into network namespace "/var/run/netns/755e18eb-85d2-4f4a-88d3-e6d3deb75d4d": failed to connect to localhost:15020 inside namespace c56b51e6328d16fd3656d35122a4dd05bf71dd9d85a845c55d730c981291e6f1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.644726Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:36323/debug/ndsz": EOF 2025-06-24T10:18:19.644755Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.645160Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:36323/stats/prometheus": dial tcp 127.0.0.1:36323: connect: connection refused 2025-06-24T10:18:19.670242Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-s6f4x_echo1") 2025-06-24T10:18:19.672577Z error klog an error occurred forwarding 36691 -> 15020: error forwarding port 15020 to pod efd760a22833edbfc002d96494f113f44e96070b7a9cd49c25b54c0cf2e28678, uid : port forward into network namespace "/var/run/netns/a28e4031-b76c-418b-8cb0-c9b148a63883": failed to connect to localhost:15020 inside namespace efd760a22833edbfc002d96494f113f44e96070b7a9cd49c25b54c0cf2e28678: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.674502Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-x7bdb for: Get "http://localhost:36691/debug/ndsz": EOF 2025-06-24T10:18:19.674518Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.675244Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-x7bdb for: Get "http://localhost:36691/stats/prometheus": dial tcp 127.0.0.1:36691: connect: connection refused 2025-06-24T10:18:19.675330Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-xjjkq_echo1") 2025-06-24T10:18:19.682941Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6887fb7d79-m2h9q_echo1") 2025-06-24T10:18:19.683636Z error klog an error occurred forwarding 33325 -> 15020: error forwarding port 15020 to pod be8407a51952c31a6af33786e4e4eb4bda30dfc8a4f84c2125d58929d8b44eff, uid : port forward into network namespace "/var/run/netns/d5ed3f10-3f79-47cb-81fb-ad0bfe80749e": failed to connect to localhost:15020 inside namespace be8407a51952c31a6af33786e4e4eb4bda30dfc8a4f84c2125d58929d8b44eff: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.685349Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-shxct for: Get "http://localhost:33325/debug/ndsz": EOF 2025-06-24T10:18:19.685368Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.686000Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-shxct for: Get "http://localhost:33325/stats/prometheus": dial tcp 127.0.0.1:33325: connect: connection refused 2025-06-24T10:18:19.686212Z error klog an error occurred forwarding 41639 -> 15020: error forwarding port 15020 to pod 6a4558d83cbe81139b54ccba61e3cf2f68e06185192bf831070072bc2b35bb21, uid : port forward into network namespace "/var/run/netns/0eb01ea0-e2f6-42a5-b186-a423fdf75f60": failed to connect to localhost:15020 inside namespace 6a4558d83cbe81139b54ccba61e3cf2f68e06185192bf831070072bc2b35bb21: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.687670Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-r488f_echo1") 2025-06-24T10:18:19.687670Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.687679Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-4q4sz for: Get "http://localhost:41639/debug/ndsz": EOF 2025-06-24T10:18:19.688440Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-4q4sz for: Get "http://localhost:41639/stats/prometheus": dial tcp 127.0.0.1:41639: connect: connection refused 2025-06-24T10:18:19.691074Z error klog an error occurred forwarding 40623 -> 15020: error forwarding port 15020 to pod 9219b26d84313691cb68ed01fac113d6565c42ac4b279754d085923d07a9965a, uid : port forward into network namespace "/var/run/netns/c8fe46a1-a7e5-4518-87d1-907b62af40a8": failed to connect to localhost:15020 inside namespace 9219b26d84313691cb68ed01fac113d6565c42ac4b279754d085923d07a9965a: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.692688Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.692777Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-dxk6g for: Get "http://localhost:40623/debug/ndsz": EOF 2025-06-24T10:18:19.693876Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-dxk6g for: Get "http://localhost:40623/stats/prometheus": dial tcp 127.0.0.1:40623: connect: connection refused 2025-06-24T10:18:19.695497Z error klog an error occurred forwarding 41469 -> 15020: error forwarding port 15020 to pod 7f047329b2410823cfbf4cc5f22ee42cb73448dfaf3ffd46ae30d3774dfdaf5b, uid : port forward into network namespace "/var/run/netns/c6fd4d93-be43-40e4-9f16-9ba150480af2": failed to connect to localhost:15020 inside namespace 7f047329b2410823cfbf4cc5f22ee42cb73448dfaf3ffd46ae30d3774dfdaf5b: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.697162Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.697195Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-9nzhq for: Get "http://localhost:41469/debug/ndsz": EOF 2025-06-24T10:18:19.698344Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-9nzhq for: Get "http://localhost:41469/stats/prometheus": dial tcp 127.0.0.1:41469: connect: connection refused 2025-06-24T10:18:19.712081Z error klog an error occurred forwarding 33503 -> 15020: error forwarding port 15020 to pod 881dbfc1b37538d3232f5ebddf62c2ea243384a4e8db1fb66c616076a6c90cef, uid : port forward into network namespace "/var/run/netns/51dcca1f-4bf1-46d7-928b-e878fb5d40f4": failed to connect to localhost:15020 inside namespace 881dbfc1b37538d3232f5ebddf62c2ea243384a4e8db1fb66c616076a6c90cef: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.713625Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-gspsg for: Get "http://localhost:33503/debug/ndsz": EOF 2025-06-24T10:18:19.713794Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.714343Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-gspsg for: Get "http://localhost:33503/stats/prometheus": dial tcp 127.0.0.1:33503: connect: connection refused 2025-06-24T10:18:19.725615Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:15Z/4074368327-8925-4c42-a09d-f1d02a5eab6e 2025-06-24T10:18:19.725714Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2219s 2025-06-24T10:18:19.726021Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-s6f4x_echo1") 2025-06-24T10:18:19.726272Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:18:19.728000Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-xjjkq_echo1") 2025-06-24T10:18:19.728537Z error klog an error occurred forwarding 41283 -> 15020: error forwarding port 15020 to pod 3b2eea046cdb1f916f344f96bbb8d6961b5647aef9d0b4afd229e0755feb6cfc, uid : port forward into network namespace "/var/run/netns/4744c60a-6384-41b5-8dcc-d20a11779018": failed to connect to localhost:15020 inside namespace 3b2eea046cdb1f916f344f96bbb8d6961b5647aef9d0b4afd229e0755feb6cfc: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:18:19.730040Z error port forward failed: lost connection to pod 2025-06-24T10:18:19.730093Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-9btd8 for: Get "http://localhost:41283/debug/ndsz": EOF 2025-06-24T10:18:19.731054Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-9btd8 for: Get "http://localhost:41283/stats/prometheus": dial tcp 127.0.0.1:41283: connect: connection refused 2025-06-24T10:18:19.742449Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-r488f_echo1") 2025-06-24T10:18:19.790148Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-s6f4x_echo1") 2025-06-24T10:18:19.803838Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-r488f_echo1") 2025-06-24T10:18:19.803994Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-757556fc5c-xjjkq_echo1") 2025-06-24T10:18:19.818584Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:15Z/40e323787c-c458-4991-b358-d7b2193ea7d1 2025-06-24T10:18:19.818779Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3149s 2025-06-24T10:18:19.819336Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:18:19.858111Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-fdfcb6f6c-s6f4x_echo1") 2025-06-24T10:18:19.872641Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-6b6b79fbc9-r488f_echo1") 2025-06-24T10:18:19.898977Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:18:19.911478Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:15Z/40c4351038-22b4-4f98-b4af-186d20c671f1 2025-06-24T10:18:19.911632Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4078s 2025-06-24T10:18:19.912304Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:18:20.004987Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-8996df688-kbsqm/istio-proxy for: pods "proxyless-grpc-v1-8996df688-kbsqm" not found 2025-06-24T10:18:20.009563Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-8996df688-kbsqm/app for: pods "proxyless-grpc-v1-8996df688-kbsqm" not found 2025-06-24T10:18:20.055979Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-v1-6887fb7d79-m2h9q/app for: pods "server-naked-foo-v1-6887fb7d79-m2h9q" not found 2025-06-24T10:18:20.605675Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:18:20.605748Z info tf === SUCCEEDED: Cleanup Istio in 68.245µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:http-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3427689 2025-06-24T10:18:31.580964Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:18:31.581025Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:18:31.612896Z info tf === Test Framework Settings === 2025-06-24T10:18:31.613000Z info tf TestID: security_ca_custom_root RunID: f2139120-5d58-44ca-a820-e8251754bae1 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:18:31.613006Z info tf =============================== 2025-06-24T10:18:31.613167Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-f213912 2025-06-24T10:18:31.613243Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:18:31.613255Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:18:31.613273Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:18:31.613280Z info tf === BEGIN: Building clusters === 2025-06-24T10:18:31.618197Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:18:31.618252Z info tf === DONE: Building clusters === 2025-06-24T10:18:31.618276Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:18:31.657841Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:18:31.657860Z info tf === SUCCESS: context setup function === 2025-06-24T10:18:31.657867Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:18:31.657880Z info tf === Istio Component Config === 2025-06-24T10:18:31.657911Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:18:31.657917Z info tf ================================ 2025-06-24T10:18:31.660938Z info tf skipping deployment as specified in the config 2025-06-24T10:18:31.660951Z info tf === SUCCEEDED: Deploy Istio in 3.082232ms [Suite=security_ca_custom_root]=== 2025-06-24T10:18:31.660958Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:18:31.746256Z info tf === SUCCEEDED: Create namespace echo1 in 85.292106ms === Generating RSA private key, 2048 bit long modulus (2 primes) *************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-f213912/_suite_context/ca-custom-root-4060065565/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************************************************************************************************************************************************************************************************+++++ ****************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-f213912/_suite_context/ca-custom-root-4060065565/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *********+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-f213912/_suite_context/ca-custom-root-4060065565/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:18:32.933985Z info tf === BEGIN: Create namespace external === 2025-06-24T10:18:32.989865Z info tf === SUCCEEDED: Create namespace external in 55.898353ms === 2025-06-24T10:18:33.321729Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:18:34.490170Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.525491Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.530168Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.537416Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.562246Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.590517Z info klog Caches are synced for echo 2025-06-24T10:18:34.616370Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.616591Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.623184Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.626119Z info klog Caches are synced for echo 2025-06-24T10:18:34.626862Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.630245Z info klog Caches are synced for echo 2025-06-24T10:18:34.638499Z info klog Caches are synced for echo 2025-06-24T10:18:34.649975Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.662805Z info klog Caches are synced for echo 2025-06-24T10:18:34.672224Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.673890Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.679061Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.679475Z info klog Waiting for caches to sync for echo 2025-06-24T10:18:34.717331Z info klog Caches are synced for echo 2025-06-24T10:18:34.717398Z info klog Caches are synced for echo 2025-06-24T10:18:34.724026Z info klog Caches are synced for echo 2025-06-24T10:18:34.727725Z info klog Caches are synced for echo 2025-06-24T10:18:34.750579Z info klog Caches are synced for echo 2025-06-24T10:18:34.773181Z info klog Caches are synced for echo 2025-06-24T10:18:34.774701Z info klog Caches are synced for echo 2025-06-24T10:18:34.780098Z info klog Caches are synced for echo 2025-06-24T10:18:34.780130Z info klog Caches are synced for echo 2025-06-24T10:18:55.484878Z info tf === SUCCEEDED: Deploy echo instances in 22.163138058s === 2025-06-24T10:18:55.514849Z info tf === DONE: Setup: 'security_ca_custom_root' (23.896561712s) === 2025-06-24T10:18:55.514890Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-06-24T10:18:55.535260Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:19:25.536019Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000754234s)' === 2025-06-24T10:19:25.536102Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:19:25.536111Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:19:25.536155Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:19:25.878004Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/491bdc585c-2f0b-4419-991f-c194b50c0859 2025-06-24T10:19:25.878398Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3421s 2025-06-24T10:19:25.878901Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:26.048371Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/495f55e95e-189b-4e83-94a1-0a54b4c719e6 2025-06-24T10:19:26.048581Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5122s 2025-06-24T10:19:26.049288Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:26.174397Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4975262ce6-cfe1-42a4-a884-64f07b233b5b 2025-06-24T10:19:26.174568Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6382s 2025-06-24T10:19:26.175140Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (31.18s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 1) (31.18s) === RUN TestTrustDomainValidation/From_primary-0 2025-06-24T10:18:55.534280Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:19:26.712998Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (31.178715768s)' === 2025-06-24T10:19:26.713078Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:19:26.713096Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:19:26.713093Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:19:27.216184Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/49244bac22-42d0-4b28-978f-0157e70d831a 2025-06-24T10:19:27.216642Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5033s 2025-06-24T10:19:27.217124Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:27.321518Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4915a1b3fb-0225-4481-9697-e5b61c0a5e75 2025-06-24T10:19:27.321666Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6084s 2025-06-24T10:19:27.322266Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:27.417191Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4917560590-429c-46a0-bd3d-67ba090bc097 2025-06-24T10:19:27.417310Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7040s 2025-06-24T10:19:27.417896Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (32.33s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (re-run 1) (32.33s) === RUN TestTrustDomainValidation 2025-06-24T10:18:55.515499Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:19:27.859696Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (32.344188797s)' === 2025-06-24T10:19:27.859831Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:19:27.859846Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:19:27.859824Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:19:28.044510Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4945fd248e-0d79-4058-bee9-0cb5a3a42ddf 2025-06-24T10:19:28.044716Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1846s 2025-06-24T10:19:28.045137Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:28.189161Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/490a4ee036-5efb-41c5-b7d7-de03f0ee1b88 2025-06-24T10:19:28.189264Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3291s 2025-06-24T10:19:28.190096Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:28.306787Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/49c217e7bc-1a08-44e9-abab-4672d4b84fba 2025-06-24T10:19:28.306924Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4468s 2025-06-24T10:19:28.307589Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (33.54s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (re-run 1) (33.54s) 2025-06-24T10:19:29.052486Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-06-24T10:19:29.052550Z info tf === Suite "security_ca_custom_root" run time: 57.43428155s === 2025-06-24T10:19:29.052570Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-06-24T10:19:29.052962Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:29.052980Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:29.052975Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:29.351470Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:29Z/502c2ef22d-8b02-404c-8fe3-320cf0bbfd3f 2025-06-24T10:19:29.351726Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2984s 2025-06-24T10:19:29.352346Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:29.512783Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:29Z/50fdbad7ae-9cb6-4b69-8afe-793061440830 2025-06-24T10:19:29.512939Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4596s 2025-06-24T10:19:29.513649Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:29.588264Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:29Z/500abe8808-e7f0-4e98-bed1-b13d3a40a6ef 2025-06-24T10:19:29.588445Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5351s 2025-06-24T10:19:29.588960Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:30.564705Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:19:30.564834Z info tf === SUCCEEDED: Cleanup Istio in 109.887µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:tcp-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3429165 2025-06-24T10:19:41.394438Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:19:41.394488Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:19:41.424899Z info tf === Test Framework Settings === 2025-06-24T10:19:41.425026Z info tf TestID: security_ca_custom_root RunID: 6167540e-5122-411c-af32-0571caf6aad8 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:19:41.425034Z info tf =============================== 2025-06-24T10:19:41.425288Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540 2025-06-24T10:19:41.425401Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:19:41.425416Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:19:41.425439Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:19:41.425451Z info tf === BEGIN: Building clusters === 2025-06-24T10:19:41.428785Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:19:41.428797Z info tf === DONE: Building clusters === 2025-06-24T10:19:41.428824Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:19:41.511547Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:19:41.511570Z info tf === SUCCESS: context setup function === 2025-06-24T10:19:41.511577Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:19:41.511593Z info tf === Istio Component Config === 2025-06-24T10:19:41.511626Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:19:41.511631Z info tf ================================ 2025-06-24T10:19:41.514436Z info tf skipping deployment as specified in the config 2025-06-24T10:19:41.514450Z info tf === SUCCEEDED: Deploy Istio in 2.870827ms [Suite=security_ca_custom_root]=== 2025-06-24T10:19:41.514456Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:19:41.553328Z info tf === SUCCEEDED: Create namespace echo1 in 38.863677ms === Generating RSA private key, 2048 bit long modulus (2 primes) *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540/_suite_context/ca-custom-root-402958698/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********+++++ ****************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540/_suite_context/ca-custom-root-402958698/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ****************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540/_suite_context/ca-custom-root-402958698/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:19:43.243399Z info tf === BEGIN: Create namespace external === 2025-06-24T10:19:43.253907Z info tf === SUCCEEDED: Create namespace external in 10.526373ms === 2025-06-24T10:19:43.383349Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:19:43.421781Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:19:43.421874Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540/_suite_context/Service.2764626035.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:19:43.421932Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:43.421936Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:43.421970Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:43.518852Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6887fb7d79-fqrvp_echo1") 2025-06-24T10:19:43.541338Z error klog an error occurred forwarding 42195 -> 15020: error forwarding port 15020 to pod 45bb7438bda7c3c6c9d0a53c5e85c31f8267457495d25c480e428f30cb715a42, uid : port forward into network namespace "/var/run/netns/f5b21255-a156-4f9b-a2f2-42715b35ae58": failed to connect to localhost:15020 inside namespace 45bb7438bda7c3c6c9d0a53c5e85c31f8267457495d25c480e428f30cb715a42: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:43.543046Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-wvbkv for: Get "http://localhost:42195/debug/ndsz": EOF 2025-06-24T10:19:43.543033Z error port forward failed: lost connection to pod 2025-06-24T10:19:43.543299Z error klog an error occurred forwarding 41243 -> 15020: error forwarding port 15020 to pod 9dc04bfa0d9b13fa7e6c75b3d4acf615523a0aeb4087ad873d2ff16d521a41e2, uid : port forward into network namespace "/var/run/netns/02659eef-b16d-4cb7-9c97-d181fba0e15f": failed to connect to localhost:15020 inside namespace 9dc04bfa0d9b13fa7e6c75b3d4acf615523a0aeb4087ad873d2ff16d521a41e2: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:43.544001Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-wvbkv for: Get "http://localhost:42195/stats/prometheus": dial tcp 127.0.0.1:42195: connect: connection refused 2025-06-24T10:19:43.544927Z error port forward failed: lost connection to pod 2025-06-24T10:19:43.544931Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:41243/debug/ndsz": EOF 2025-06-24T10:19:43.545789Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:41243/stats/prometheus": dial tcp 127.0.0.1:41243: connect: connection refused 2025-06-24T10:19:43.555141Z error klog an error occurred forwarding 44545 -> 15020: error forwarding port 15020 to pod 40fb6f6c0c1c364634156beda437535bf1eeff9f3440e00f2130bc43763892c8, uid : port forward into network namespace "/var/run/netns/2639c90f-ffda-43d3-8ec9-15fce21e9f00": failed to connect to localhost:15020 inside namespace 40fb6f6c0c1c364634156beda437535bf1eeff9f3440e00f2130bc43763892c8: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:43.557023Z error port forward failed: lost connection to pod 2025-06-24T10:19:43.557038Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-5vrfv for: Get "http://localhost:44545/debug/ndsz": EOF 2025-06-24T10:19:43.558069Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-5vrfv for: Get "http://localhost:44545/stats/prometheus": dial tcp 127.0.0.1:44545: connect: connection refused 2025-06-24T10:19:43.564010Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:43Z/541532a83d-bde0-4fca-8732-4dd0e433454f 2025-06-24T10:19:43.564227Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1418s 2025-06-24T10:19:43.564825Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:43.579749Z error klog an error occurred forwarding 40637 -> 15020: error forwarding port 15020 to pod 41bdd898dfdd6e70418ef00e66d34c06a88c0641ddfc40b8441c934665d2e7de, uid : port forward into network namespace "/var/run/netns/f2574f90-a969-4a83-811d-fddfddda02d7": failed to connect to localhost:15020 inside namespace 41bdd898dfdd6e70418ef00e66d34c06a88c0641ddfc40b8441c934665d2e7de: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:43.581547Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-8b9c9 for: Get "http://localhost:40637/debug/ndsz": EOF 2025-06-24T10:19:43.581611Z error port forward failed: lost connection to pod 2025-06-24T10:19:43.582174Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-8b9c9 for: Get "http://localhost:40637/stats/prometheus": dial tcp 127.0.0.1:40637: connect: connection refused 2025-06-24T10:19:43.599934Z error klog an error occurred forwarding 34305 -> 15020: error forwarding port 15020 to pod 52fa768306c400f26b734e2b1d157d1a4b11ea1a8fd7557f2be7ecfe1860f7f1, uid : port forward into network namespace "/var/run/netns/3c9ccc28-3ec6-4128-8091-9dccca714e4d": failed to connect to localhost:15020 inside namespace 52fa768306c400f26b734e2b1d157d1a4b11ea1a8fd7557f2be7ecfe1860f7f1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:43.602282Z error port forward failed: lost connection to pod 2025-06-24T10:19:43.602293Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-tfb42 for: Get "http://localhost:34305/debug/ndsz": EOF 2025-06-24T10:19:43.603384Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-tfb42 for: Get "http://localhost:34305/stats/prometheus": dial tcp 127.0.0.1:34305: connect: connection refused 2025-06-24T10:19:43.640358Z error klog an error occurred forwarding 44625 -> 15020: error forwarding port 15020 to pod 67830e03f87b999eff9fb388a7bd630ed610896ec7936bd87fe8c9955c22f83d, uid : port forward into network namespace "/var/run/netns/6d2cfad4-27b2-4099-9416-3fa660d3162b": failed to connect to localhost:15020 inside namespace 67830e03f87b999eff9fb388a7bd630ed610896ec7936bd87fe8c9955c22f83d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:43.642717Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:44625/debug/ndsz": EOF 2025-06-24T10:19:43.642897Z error port forward failed: lost connection to pod 2025-06-24T10:19:43.643839Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:44625/stats/prometheus": dial tcp 127.0.0.1:44625: connect: connection refused 2025-06-24T10:19:43.744363Z error klog an error occurred forwarding 43415 -> 15020: error forwarding port 15020 to pod 6a31a6361ae8b5d58572901912495d68b2bf11991fb90f3c504851177b718acc, uid : port forward into network namespace "/var/run/netns/a498244e-15eb-4179-827d-593c4ceb3c19": failed to connect to localhost:15020 inside namespace 6a31a6361ae8b5d58572901912495d68b2bf11991fb90f3c504851177b718acc: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:43.746595Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-f7698 for: Get "http://localhost:43415/debug/ndsz": EOF 2025-06-24T10:19:43.746617Z error port forward failed: lost connection to pod 2025-06-24T10:19:43.747355Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-f7698 for: Get "http://localhost:43415/stats/prometheus": dial tcp 127.0.0.1:43415: connect: connection refused 2025-06-24T10:19:43.747379Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:19:43.776254Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:43Z/544c21340a-2f4f-44bb-9d8e-62b1a1e3563f 2025-06-24T10:19:43.776380Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3540s 2025-06-24T10:19:43.777092Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:43.956684Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:43Z/54598d75b3-788f-4aef-8562-0f9aebec25f9 2025-06-24T10:19:43.956856Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5344s 2025-06-24T10:19:43.957526Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:44.328673Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/naked-v1-757556fc5c-fh6pw/app for: pods "naked-v1-757556fc5c-fh6pw" not found 2025-06-24T10:19:44.335255Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-8996df688-q6zxr/istio-proxy for: pods "proxyless-grpc-v1-8996df688-q6zxr" not found 2025-06-24T10:19:44.344877Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-8996df688-q6zxr/app for: pods "proxyless-grpc-v1-8996df688-q6zxr" not found 2025-06-24T10:19:44.353881Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-bar-v1-fdfcb6f6c-f9bsf/app for: pods "server-naked-bar-v1-fdfcb6f6c-f9bsf" not found 2025-06-24T10:19:44.366587Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-alt-v1-6b6b79fbc9-4v45c/app for: pods "server-naked-foo-alt-v1-6b6b79fbc9-4v45c" not found 2025-06-24T10:19:44.372944Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-v1-6887fb7d79-fqrvp/app for: pods "server-naked-foo-v1-6887fb7d79-fqrvp" not found 2025-06-24T10:19:46.290654Z error klog an error occurred forwarding 37217 -> 15000: error forwarding port 15000 to pod 40fb6f6c0c1c364634156beda437535bf1eeff9f3440e00f2130bc43763892c8, uid : port forward into network namespace "/var/run/netns/2639c90f-ffda-43d3-8ec9-15fce21e9f00": writeto tcp 127.0.0.1:43010->127.0.0.1:15000: read tcp 127.0.0.1:43010->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.292968Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.292985Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-5vrfv for: Get "http://localhost:37217/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.293805Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-5vrfv for: Get "http://localhost:37217/clusters": dial tcp 127.0.0.1:37217: connect: connection refused 2025-06-24T10:19:46.322140Z error klog an error occurred forwarding 35649 -> 15000: error forwarding port 15000 to pod 6a31a6361ae8b5d58572901912495d68b2bf11991fb90f3c504851177b718acc, uid : port forward into network namespace "/var/run/netns/a498244e-15eb-4179-827d-593c4ceb3c19": writeto tcp 127.0.0.1:36022->127.0.0.1:15000: read tcp 127.0.0.1:36022->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.324249Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-f7698 for: Get "http://localhost:35649/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.324281Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.325212Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-f7698 for: Get "http://localhost:35649/clusters": dial tcp 127.0.0.1:35649: connect: connection refused 2025-06-24T10:19:46.353660Z error klog an error occurred forwarding 36773 -> 15000: error forwarding port 15000 to pod 45bb7438bda7c3c6c9d0a53c5e85c31f8267457495d25c480e428f30cb715a42, uid : port forward into network namespace "/var/run/netns/f5b21255-a156-4f9b-a2f2-42715b35ae58": writeto tcp 127.0.0.1:52196->127.0.0.1:15000: read tcp 127.0.0.1:52196->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.355624Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.355628Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-wvbkv for: Get "http://localhost:36773/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.356711Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-wvbkv for: Get "http://localhost:36773/clusters": dial tcp 127.0.0.1:36773: connect: connection refused 2025-06-24T10:19:46.409608Z error klog an error occurred forwarding 44295 -> 15000: error forwarding port 15000 to pod 9dc04bfa0d9b13fa7e6c75b3d4acf615523a0aeb4087ad873d2ff16d521a41e2, uid : port forward into network namespace "/var/run/netns/02659eef-b16d-4cb7-9c97-d181fba0e15f": writeto tcp 127.0.0.1:52176->127.0.0.1:15000: read tcp 127.0.0.1:52176->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.412073Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:44295/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.412162Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.413046Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:44295/clusters": dial tcp 127.0.0.1:44295: connect: connection refused 2025-06-24T10:19:46.485069Z error klog an error occurred forwarding 43461 -> 15000: error forwarding port 15000 to pod 67830e03f87b999eff9fb388a7bd630ed610896ec7936bd87fe8c9955c22f83d, uid : port forward into network namespace "/var/run/netns/6d2cfad4-27b2-4099-9416-3fa660d3162b": writeto tcp 127.0.0.1:36042->127.0.0.1:15000: read tcp 127.0.0.1:36042->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.487606Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:43461/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.487658Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.488358Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:43461/clusters": dial tcp 127.0.0.1:43461: connect: connection refused 2025-06-24T10:19:46.568034Z error klog an error occurred forwarding 43943 -> 15000: error forwarding port 15000 to pod 52fa768306c400f26b734e2b1d157d1a4b11ea1a8fd7557f2be7ecfe1860f7f1, uid : port forward into network namespace "/var/run/netns/3c9ccc28-3ec6-4128-8091-9dccca714e4d": writeto tcp 127.0.0.1:43020->127.0.0.1:15000: read tcp 127.0.0.1:43020->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.570374Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-tfb42 for: Get "http://localhost:43943/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.570461Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.571226Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-tfb42 for: Get "http://localhost:43943/clusters": dial tcp 127.0.0.1:43943: connect: connection refused 2025-06-24T10:19:46.631647Z error klog an error occurred forwarding 39955 -> 15000: error forwarding port 15000 to pod 41bdd898dfdd6e70418ef00e66d34c06a88c0641ddfc40b8441c934665d2e7de, uid : port forward into network namespace "/var/run/netns/f2574f90-a969-4a83-811d-fddfddda02d7": writeto tcp 127.0.0.1:36056->127.0.0.1:15000: read tcp 127.0.0.1:36056->127.0.0.1:15000: read: connection reset by peer: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.634091Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.634096Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-8b9c9 for: Get "http://localhost:39955/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.634895Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-8b9c9 for: Get "http://localhost:39955/clusters": dial tcp 127.0.0.1:39955: connect: connection refused 2025-06-24T10:19:46.640033Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540/_suite_context/Service.2764626035.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:19:46.640063Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540/_suite_context/Service.2764626035.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-06-24T10:19:46.640073Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6167540/_suite_context/Service.2764626035.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:19:46.640110Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:46.640114Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:46.640143Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:19:46.765320Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-65f898ccd5-8b9c9_echo1") 2025-06-24T10:19:46.769061Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-7bcb76fc6f-5vrfv_echo1") 2025-06-24T10:19:46.787728Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-65f898ccd5-8b9c9_echo1") 2025-06-24T10:19:46.797313Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-86cfb5b889-tfb42_echo1") 2025-06-24T10:19:46.799544Z error klog an error occurred forwarding 33949 -> 15020: error forwarding port 15020 to pod 67830e03f87b999eff9fb388a7bd630ed610896ec7936bd87fe8c9955c22f83d, uid : port forward into network namespace "/var/run/netns/6d2cfad4-27b2-4099-9416-3fa660d3162b": failed to connect to localhost:15020 inside namespace 67830e03f87b999eff9fb388a7bd630ed610896ec7936bd87fe8c9955c22f83d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.801689Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.801698Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:33949/debug/ndsz": EOF 2025-06-24T10:19:46.802462Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:33949/stats/prometheus": dial tcp 127.0.0.1:33949: connect: connection refused 2025-06-24T10:19:46.808870Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-wvbkv_echo1") 2025-06-24T10:19:46.814229Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-78f49b749b-wvbkv_echo1") 2025-06-24T10:19:46.816839Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-86cfb5b889-tfb42_echo1") 2025-06-24T10:19:46.827246Z error klog an error occurred forwarding 33293 -> 15000: error forwarding port 15000 to pod 67830e03f87b999eff9fb388a7bd630ed610896ec7936bd87fe8c9955c22f83d, uid : sandbox 67830e03f87b999eff9fb388a7bd630ed610896ec7936bd87fe8c9955c22f83d is not running: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.829187Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.829212Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:33293/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.831020Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-pd8lw for: Get "http://localhost:33293/clusters": dial tcp 127.0.0.1:33293: connect: connection refused 2025-06-24T10:19:46.831846Z error klog an error occurred forwarding 46879 -> 15020: error forwarding port 15020 to pod 9dc04bfa0d9b13fa7e6c75b3d4acf615523a0aeb4087ad873d2ff16d521a41e2, uid : port forward into network namespace "/var/run/netns/02659eef-b16d-4cb7-9c97-d181fba0e15f": failed to connect to localhost:15020 inside namespace 9dc04bfa0d9b13fa7e6c75b3d4acf615523a0aeb4087ad873d2ff16d521a41e2: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.833302Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:46879/debug/ndsz": EOF 2025-06-24T10:19:46.833318Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.833932Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:46879/stats/prometheus": dial tcp 127.0.0.1:46879: connect: connection refused 2025-06-24T10:19:46.836983Z error klog an error occurred forwarding 42463 -> 15000: error forwarding port 15000 to pod 9dc04bfa0d9b13fa7e6c75b3d4acf615523a0aeb4087ad873d2ff16d521a41e2, uid : port forward into network namespace "/var/run/netns/02659eef-b16d-4cb7-9c97-d181fba0e15f": failed to connect to localhost:15000 inside namespace 9dc04bfa0d9b13fa7e6c75b3d4acf615523a0aeb4087ad873d2ff16d521a41e2: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:19:46.838834Z error port forward failed: lost connection to pod 2025-06-24T10:19:46.838874Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:42463/config_dump?include_eds=true": EOF 2025-06-24T10:19:46.839830Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-qlrgh for: Get "http://localhost:42463/clusters": dial tcp 127.0.0.1:42463: connect: connection refused 2025-06-24T10:19:46.849285Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-7bcb76fc6f-5vrfv_echo1") 2025-06-24T10:19:46.867562Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:44Z/550e2e9bca-6c86-4ffb-a9f7-65e84bd514c8 2025-06-24T10:19:46.867654Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2271s 2025-06-24T10:19:46.868405Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:46.934148Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-86cfb5b889-tfb42_echo1") 2025-06-24T10:19:46.950314Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:19:47.033158Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/c-v1-6975698f5d-f7698/istio-validation for: pods "c-v1-6975698f5d-f7698" not found 2025-06-24T10:19:47.057745Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:19:47.076912Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:44Z/556cfe84e2-47b5-4909-9fef-e5e7a797b297 2025-06-24T10:19:47.077098Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4365s 2025-06-24T10:19:47.077847Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:47.179040Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:19:44Z/5588545931-b843-4394-91ed-57c481428686 2025-06-24T10:19:47.179292Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5387s 2025-06-24T10:19:47.180004Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:47.202057Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-8b9c9/app for: pods "sotw-v1-65f898ccd5-8b9c9" not found 2025-06-24T10:19:47.204764Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-8b9c9/istio-proxy for: pods "sotw-v1-65f898ccd5-8b9c9" not found 2025-06-24T10:19:47.206881Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-8b9c9/istio-validation for: pods "sotw-v1-65f898ccd5-8b9c9" not found 2025-06-24T10:19:47.211396Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:19:47.211415Z info tf === SUCCEEDED: Cleanup Istio in 15.89µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:tcp-mtls-pass-through\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3430452 2025-06-24T10:19:58.384285Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:19:58.384337Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:19:58.415552Z info tf === Test Framework Settings === 2025-06-24T10:19:58.415699Z info tf TestID: security_ca_custom_root RunID: 695572c6-36b6-472c-b186-514e3559c176 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:19:58.415707Z info tf =============================== 2025-06-24T10:19:58.415933Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-695572c 2025-06-24T10:19:58.416039Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:19:58.416061Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:19:58.416083Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:19:58.416094Z info tf === BEGIN: Building clusters === 2025-06-24T10:19:58.421428Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:19:58.421478Z info tf === DONE: Building clusters === 2025-06-24T10:19:58.421504Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:19:58.465075Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:19:58.465150Z info tf === SUCCESS: context setup function === 2025-06-24T10:19:58.465161Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:19:58.465184Z info tf === Istio Component Config === 2025-06-24T10:19:58.465255Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:19:58.465264Z info tf ================================ 2025-06-24T10:19:58.469208Z info tf skipping deployment as specified in the config 2025-06-24T10:19:58.469225Z info tf === SUCCEEDED: Deploy Istio in 4.061629ms [Suite=security_ca_custom_root]=== 2025-06-24T10:19:58.469232Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:19:58.552188Z info tf === SUCCEEDED: Create namespace echo1 in 82.948236ms === Generating RSA private key, 2048 bit long modulus (2 primes) ****************************************************************************+++++ **************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-695572c/_suite_context/ca-custom-root-1307589800/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ******************************************************************************************************************************************************************************************************************************************************************+++++ *******************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-695572c/_suite_context/ca-custom-root-1307589800/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ******************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-695572c/_suite_context/ca-custom-root-1307589800/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:20:00.022971Z info tf === BEGIN: Create namespace external === 2025-06-24T10:20:00.051628Z info tf === SUCCEEDED: Create namespace external in 28.666655ms === 2025-06-24T10:20:00.244559Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:20:01.664224Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.667314Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.676203Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.724937Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.749090Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.754426Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.762060Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.765110Z info klog Caches are synced for echo 2025-06-24T10:20:01.767945Z info klog Caches are synced for echo 2025-06-24T10:20:01.776451Z info klog Caches are synced for echo 2025-06-24T10:20:01.778820Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.805874Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.806537Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.825195Z info klog Caches are synced for echo 2025-06-24T10:20:01.826198Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.849367Z info klog Caches are synced for echo 2025-06-24T10:20:01.852952Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.855341Z info klog Caches are synced for echo 2025-06-24T10:20:01.857824Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.860648Z info klog Waiting for caches to sync for echo 2025-06-24T10:20:01.862299Z info klog Caches are synced for echo 2025-06-24T10:20:01.879597Z info klog Caches are synced for echo 2025-06-24T10:20:01.906177Z info klog Caches are synced for echo 2025-06-24T10:20:01.907620Z info klog Caches are synced for echo 2025-06-24T10:20:01.927292Z info klog Caches are synced for echo 2025-06-24T10:20:01.953199Z info klog Caches are synced for echo 2025-06-24T10:20:01.957992Z info klog Caches are synced for echo 2025-06-24T10:20:01.961430Z info klog Caches are synced for echo 2025-06-24T10:20:30.619535Z info tf === SUCCEEDED: Deploy echo instances in 30.374967476s === 2025-06-24T10:20:30.689190Z info tf === DONE: Setup: 'security_ca_custom_root' (32.267665645s) === 2025-06-24T10:20:30.689266Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] 2025-06-24T10:20:30.738992Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.21.3.142:9000': call failed from naked (cluster=cluster-0) to tcp://172.21.3.142:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:21:00.739881Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.00088158s)' === 2025-06-24T10:21:00.740022Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:21:00.739988Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:21:00.740012Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:21:01.063645Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68aa30caef-f0b3-41f1-91a2-e9c61c11a8df 2025-06-24T10:21:01.063917Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3231s 2025-06-24T10:21:01.064547Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:01.155390Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68c9414860-ead5-4dcf-9033-490578e0519e 2025-06-24T10:21:01.155670Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4148s 2025-06-24T10:21:01.156307Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:01.235100Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68a7d86e13-eca7-4b06-bcef-db8a671bb7b1 2025-06-24T10:21:01.235249Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4944s 2025-06-24T10:21:01.235986Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.18s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 1) (31.18s) === RUN TestTrustDomainValidation/From_primary-0 2025-06-24T10:20:30.738126Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:21:01.921068Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (31.182935092s)' === 2025-06-24T10:21:01.921196Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:21:01.921198Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:21:01.921222Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:21:02.111299Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/680ee68527-6a9f-4c50-9b8d-40feef3c56c5 2025-06-24T10:21:02.111613Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1902s 2025-06-24T10:21:02.112013Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:02.199728Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/6897e73ddf-d3a4-4a02-9183-03b10b716d0e 2025-06-24T10:21:02.199910Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2785s 2025-06-24T10:21:02.200607Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:02.349467Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68bfac5bbb-fa1a-49e2-bff9-084bdbc138e4 2025-06-24T10:21:02.349680Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4282s 2025-06-24T10:21:02.351904Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (32.24s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (re-run 1) (32.24s) === RUN TestTrustDomainValidation 2025-06-24T10:20:30.689905Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:21:02.981236Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (32.291319037s)' === 2025-06-24T10:21:02.981379Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:21:02.981397Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:21:02.981387Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:21:03.219915Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/6880b502a3-ce16-45b2-9e75-8ec0ac0695b3 2025-06-24T10:21:03.220222Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2386s 2025-06-24T10:21:03.220598Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:03.345260Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68506004ce-ad2e-4d4f-aea4-b554a34bc2fd 2025-06-24T10:21:03.345401Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3637s 2025-06-24T10:21:03.345990Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:03.442976Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/683e933305-9c3f-4b98-9113-c771256984d5 2025-06-24T10:21:03.443114Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4614s 2025-06-24T10:21:03.443685Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (33.42s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (re-run 1) (33.42s) 2025-06-24T10:21:04.112471Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-06-24T10:21:04.112523Z info tf === Suite "security_ca_custom_root" run time: 1m5.691028518s === 2025-06-24T10:21:04.112535Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-06-24T10:21:04.112743Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:04.112750Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:04.112789Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:04.375940Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:04Z/69d2020c78-6f4d-43ac-bde5-8f3742d4b47d 2025-06-24T10:21:04.376238Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2631s 2025-06-24T10:21:04.376718Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:04.493793Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:04Z/6996876b71-db97-4113-8c29-be62be2960e0 2025-06-24T10:21:04.494182Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3811s 2025-06-24T10:21:04.494838Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:04.599011Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:04Z/6998b2477e-bdf0-44f6-b13b-6ba3d815ce96 2025-06-24T10:21:04.599258Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4861s 2025-06-24T10:21:04.599869Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:05.781935Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:21:05.782047Z info tf === SUCCEEDED: Cleanup Istio in 103.15µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3431963 2025-06-24T10:21:16.005117Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:21:16.005170Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:21:16.036270Z info tf === Test Framework Settings === 2025-06-24T10:21:16.036392Z info tf TestID: security_ca_custom_root RunID: 80eb53e2-2604-4f85-b74d-0759f6cd6830 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:21:16.036401Z info tf =============================== 2025-06-24T10:21:16.036568Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e 2025-06-24T10:21:16.036644Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:21:16.036654Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:21:16.036670Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:21:16.036679Z info tf === BEGIN: Building clusters === 2025-06-24T10:21:16.039405Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:21:16.039414Z info tf === DONE: Building clusters === 2025-06-24T10:21:16.039429Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:21:16.079446Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:21:16.079516Z info tf === SUCCESS: context setup function === 2025-06-24T10:21:16.079524Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:21:16.079539Z info tf === Istio Component Config === 2025-06-24T10:21:16.079573Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:21:16.079578Z info tf ================================ 2025-06-24T10:21:16.082988Z info tf skipping deployment as specified in the config 2025-06-24T10:21:16.083004Z info tf === SUCCEEDED: Deploy Istio in 3.478803ms [Suite=security_ca_custom_root]=== 2025-06-24T10:21:16.083011Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:21:16.157611Z info tf === SUCCEEDED: Create namespace echo1 in 74.593299ms === Generating RSA private key, 2048 bit long modulus (2 primes) *******************************************************************************************************************************************************************************+++++ **************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e/_suite_context/ca-custom-root-2035536437/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***************************************************************************************************************************************************************************************************************************************************************************************+++++ ******************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e/_suite_context/ca-custom-root-2035536437/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ************************************************************************************************************+++++ ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e/_suite_context/ca-custom-root-2035536437/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:21:17.291824Z info tf === BEGIN: Create namespace external === 2025-06-24T10:21:17.319773Z info tf === SUCCEEDED: Create namespace external in 27.971025ms === 2025-06-24T10:21:17.533923Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:21:17.628281Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:21:17.628343Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e/_suite_context/ServiceAccount-Service.2316179545.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-06-24T10:21:17.628406Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:17.628398Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:17.628433Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:18.468654Z warn tf cluster/pod cluster-0/istio-system/istio-ingressgateway-bf7cd59d6-64wm6 found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:18.508433Z error klog an error occurred forwarding 34105 -> 15020: error forwarding port 15020 to pod c773a3c89fa565201eb599fc5696fe74808c2b9c913645c66d2230df3671b902, uid : port forward into network namespace "/var/run/netns/8585eab9-3e4b-4245-878c-64a4ba7fc8ad": failed to connect to localhost:15020 inside namespace c773a3c89fa565201eb599fc5696fe74808c2b9c913645c66d2230df3671b902: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.509970Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.509959Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-pcpj5 for: Get "http://localhost:34105/debug/ndsz": EOF 2025-06-24T10:21:18.510854Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-pcpj5 for: Get "http://localhost:34105/stats/prometheus": dial tcp 127.0.0.1:34105: connect: connection refused 2025-06-24T10:21:18.574965Z error klog an error occurred forwarding 34109 -> 15020: error forwarding port 15020 to pod 003696867df9c0fde06d2a7e063247177fcb1953d40e7cafc336860997233068, uid : port forward into network namespace "/var/run/netns/5837e964-3b40-4318-991b-126a22226a2b": failed to connect to localhost:15020 inside namespace 003696867df9c0fde06d2a7e063247177fcb1953d40e7cafc336860997233068: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.577092Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:34109/debug/ndsz": EOF 2025-06-24T10:21:18.577153Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.577683Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:34109/stats/prometheus": dial tcp 127.0.0.1:34109: connect: connection refused 2025-06-24T10:21:18.628297Z error klog an error occurred forwarding 38979 -> 15020: error forwarding port 15020 to pod 508bfb33fd745094733e4891007e773983895ea560aec42ab802e9a12f58f57e, uid : port forward into network namespace "/var/run/netns/e4b5b7c4-20ef-473c-b055-96eeb2492952": failed to connect to localhost:15020 inside namespace 508bfb33fd745094733e4891007e773983895ea560aec42ab802e9a12f58f57e: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.630847Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/proxyless-grpc-v1-8996df688-c6w92 for: Get "http://localhost:38979/debug/ndsz": EOF 2025-06-24T10:21:18.630867Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.631380Z error klog an error occurred forwarding 40083 -> 15020: error forwarding port 15020 to pod 71279f225ab9ec1617dd9b712081e6d9b16c76b441a3f8cd5edc70f6f6e4151e, uid : port forward into network namespace "/var/run/netns/99ef4d62-f422-4196-b676-a8b15241952a": failed to connect to localhost:15020 inside namespace 71279f225ab9ec1617dd9b712081e6d9b16c76b441a3f8cd5edc70f6f6e4151e: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.631764Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/proxyless-grpc-v1-8996df688-c6w92 for: Get "http://localhost:38979/stats/prometheus": dial tcp 127.0.0.1:38979: connect: connection refused 2025-06-24T10:21:18.633814Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.633830Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-c2trk for: Get "http://localhost:40083/debug/ndsz": EOF 2025-06-24T10:21:18.634695Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-c2trk for: Get "http://localhost:40083/stats/prometheus": dial tcp 127.0.0.1:40083: connect: connection refused 2025-06-24T10:21:18.742892Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:17Z/73a787dd92-006f-4bf0-9c44-3bb199c8ee35 2025-06-24T10:21:18.743117Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1143s 2025-06-24T10:21:18.744032Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:18.745247Z error klog an error occurred forwarding 41693 -> 15000: error forwarding port 15000 to pod 003696867df9c0fde06d2a7e063247177fcb1953d40e7cafc336860997233068, uid : port forward into network namespace "/var/run/netns/5837e964-3b40-4318-991b-126a22226a2b": failed to connect to localhost:15000 inside namespace 003696867df9c0fde06d2a7e063247177fcb1953d40e7cafc336860997233068: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.747272Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.747313Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:41693/config_dump?include_eds=true": EOF 2025-06-24T10:21:18.748455Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:41693/clusters": dial tcp 127.0.0.1:41693: connect: connection refused 2025-06-24T10:21:18.933713Z error klog an error occurred forwarding 43693 -> 15020: error forwarding port 15020 to pod be4de9244cf6721d7393666e94817b13c69582dad2e421839dfac2d563ea6515, uid : port forward into network namespace "/var/run/netns/329b243b-3143-49db-80f5-e7a46341d615": failed to connect to localhost:15020 inside namespace be4de9244cf6721d7393666e94817b13c69582dad2e421839dfac2d563ea6515: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.935909Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.935950Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-qvvzr for: Get "http://localhost:43693/debug/ndsz": EOF 2025-06-24T10:21:18.936846Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-qvvzr for: Get "http://localhost:43693/stats/prometheus": dial tcp 127.0.0.1:43693: connect: connection refused 2025-06-24T10:21:18.955346Z error klog an error occurred forwarding 38071 -> 15020: error forwarding port 15020 to pod f6aeea49f1421af6353c8df4982128f912b25c486ca0fd78198c5e88e6b6c005, uid : port forward into network namespace "/var/run/netns/286b4106-157d-4c34-a87b-f9dffcac1d95": failed to connect to localhost:15020 inside namespace f6aeea49f1421af6353c8df4982128f912b25c486ca0fd78198c5e88e6b6c005: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.956813Z error klog an error occurred forwarding 42429 -> 15020: error forwarding port 15020 to pod 6c9077d7c028cc7f7ea36db65b71ccef76c25231f60fa8cabfd9ee55cf29dff0, uid : port forward into network namespace "/var/run/netns/bc9cef9d-f0bd-47bf-bbe9-e3b6bacd553d": failed to connect to localhost:15020 inside namespace 6c9077d7c028cc7f7ea36db65b71ccef76c25231f60fa8cabfd9ee55cf29dff0: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.956824Z error klog an error occurred forwarding 46307 -> 15020: error forwarding port 15020 to pod 224b4fa0c71d346ee49f508ff32b73e1cc765f07c3e646145b2e95ffd547e4b5, uid : port forward into network namespace "/var/run/netns/6bf1c40f-8a4c-4a85-b9f4-d5061d08a70a": failed to connect to localhost:15020 inside namespace 224b4fa0c71d346ee49f508ff32b73e1cc765f07c3e646145b2e95ffd547e4b5: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.957248Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.957282Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-h26fk for: Get "http://localhost:38071/debug/ndsz": EOF 2025-06-24T10:21:18.958075Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-h26fk for: Get "http://localhost:38071/stats/prometheus": dial tcp 127.0.0.1:38071: connect: connection refused 2025-06-24T10:21:18.958419Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-rzqgk for: Get "http://localhost:46307/debug/ndsz": EOF 2025-06-24T10:21:18.958442Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-kp8w6 for: Get "http://localhost:42429/debug/ndsz": EOF 2025-06-24T10:21:18.958458Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.958494Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.959075Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-kp8w6 for: Get "http://localhost:42429/stats/prometheus": dial tcp 127.0.0.1:42429: connect: connection refused 2025-06-24T10:21:18.959271Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-rzqgk for: Get "http://localhost:46307/stats/prometheus": dial tcp 127.0.0.1:46307: connect: connection refused 2025-06-24T10:21:18.985780Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:17Z/73b76f0925-34f8-46a2-ac97-dbff9330f799 2025-06-24T10:21:18.985900Z info tf istioctl ([x internal-debug --all configz]): completed after 1.3571s 2025-06-24T10:21:18.986514Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:18.990515Z error klog an error occurred forwarding 33035 -> 15020: error forwarding port 15020 to pod 7b636f2f5405b6272599cb2c1c7fd4b84772492ecff6618acf769cc14ffe4e30, uid : port forward into network namespace "/var/run/netns/64cf4f09-0543-4672-9ab5-174741baeb44": failed to connect to localhost:15020 inside namespace 7b636f2f5405b6272599cb2c1c7fd4b84772492ecff6618acf769cc14ffe4e30: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:18.992193Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb for: Get "http://localhost:33035/debug/ndsz": EOF 2025-06-24T10:21:18.992208Z error port forward failed: lost connection to pod 2025-06-24T10:21:18.992987Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb for: Get "http://localhost:33035/stats/prometheus": dial tcp 127.0.0.1:33035: connect: connection refused 2025-06-24T10:21:19.161201Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:17Z/73c493cdc3-0b00-4099-9672-247dfe4af4b5 2025-06-24T10:21:19.161362Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.5325s 2025-06-24T10:21:19.162052Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:20.161134Z warn tf cluster/pod cluster-0/echo1/c-v1-6975698f5d-pcpj5 found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:20.384254Z warn tf cluster/pod cluster-0/echo1/client-v1-546c55dc9c-kp8w6 found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:20.505029Z warn tf cluster/pod cluster-0/echo1/headless-v1-64d65d6c8b-h26fk found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:20.587818Z warn tf cluster/pod cluster-0/echo1/b-v1-7bcb76fc6f-qvvzr found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:20.730173Z warn tf cluster/pod cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:21.213223Z warn tf cluster/pod cluster-0/echo1/a-v1-78f49b749b-rzqgk found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:21.400111Z warn tf cluster/pod cluster-0/echo1/server-v1-86cfb5b889-c2trk found warming resources (dynamic_warming_clusters) on attempt 1 2025-06-24T10:21:22.513922Z warn tf cluster/pod cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb found warming resources (dynamic_warming_clusters) on attempt 2 2025-06-24T10:21:25.232290Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e/_suite_context/ServiceAccount-Service.2316179545.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-06-24T10:21:25.232409Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e/_suite_context/ServiceAccount-Service.2316179545.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-06-24T10:21:25.232419Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-80eb53e/_suite_context/ServiceAccount-Service.2316179545.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-06-24T10:21:25.232473Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:25.232498Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:25.232485Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:25.425881Z error klog an error occurred forwarding 40345 -> 15000: error forwarding port 15000 to pod 6c9077d7c028cc7f7ea36db65b71ccef76c25231f60fa8cabfd9ee55cf29dff0, uid : port forward into network namespace "/var/run/netns/bc9cef9d-f0bd-47bf-bbe9-e3b6bacd553d": failed to connect to localhost:15000 inside namespace 6c9077d7c028cc7f7ea36db65b71ccef76c25231f60fa8cabfd9ee55cf29dff0: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.428583Z error klog an error occurred forwarding 37317 -> 15020: error forwarding port 15020 to pod 71279f225ab9ec1617dd9b712081e6d9b16c76b441a3f8cd5edc70f6f6e4151e, uid : port forward into network namespace "/var/run/netns/99ef4d62-f422-4196-b676-a8b15241952a": failed to connect to localhost:15020 inside namespace 71279f225ab9ec1617dd9b712081e6d9b16c76b441a3f8cd5edc70f6f6e4151e: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.428769Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.428797Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-kp8w6 for: Get "http://localhost:40345/config_dump?include_eds=true": EOF 2025-06-24T10:21:25.429557Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-kp8w6 for: Get "http://localhost:40345/clusters": dial tcp 127.0.0.1:40345: connect: connection refused 2025-06-24T10:21:25.430270Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-c2trk for: Get "http://localhost:37317/debug/ndsz": EOF 2025-06-24T10:21:25.430318Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.431027Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-c2trk for: Get "http://localhost:37317/stats/prometheus": dial tcp 127.0.0.1:37317: connect: connection refused 2025-06-24T10:21:25.439054Z error klog an error occurred forwarding 35299 -> 15020: error forwarding port 15020 to pod 224b4fa0c71d346ee49f508ff32b73e1cc765f07c3e646145b2e95ffd547e4b5, uid : port forward into network namespace "/var/run/netns/6bf1c40f-8a4c-4a85-b9f4-d5061d08a70a": failed to connect to localhost:15020 inside namespace 224b4fa0c71d346ee49f508ff32b73e1cc765f07c3e646145b2e95ffd547e4b5: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.440942Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-rzqgk for: Get "http://localhost:35299/debug/ndsz": EOF 2025-06-24T10:21:25.440967Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.441651Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-rzqgk for: Get "http://localhost:35299/stats/prometheus": dial tcp 127.0.0.1:35299: connect: connection refused 2025-06-24T10:21:25.449335Z error klog an error occurred forwarding 45509 -> 15020: error forwarding port 15020 to pod f6aeea49f1421af6353c8df4982128f912b25c486ca0fd78198c5e88e6b6c005, uid : port forward into network namespace "/var/run/netns/286b4106-157d-4c34-a87b-f9dffcac1d95": failed to connect to localhost:15020 inside namespace f6aeea49f1421af6353c8df4982128f912b25c486ca0fd78198c5e88e6b6c005: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.449747Z error klog an error occurred forwarding 37739 -> 15000: error forwarding port 15000 to pod c773a3c89fa565201eb599fc5696fe74808c2b9c913645c66d2230df3671b902, uid : port forward into network namespace "/var/run/netns/8585eab9-3e4b-4245-878c-64a4ba7fc8ad": failed to connect to localhost:15000 inside namespace c773a3c89fa565201eb599fc5696fe74808c2b9c913645c66d2230df3671b902: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.451000Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.450992Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-h26fk for: Get "http://localhost:45509/debug/ndsz": EOF 2025-06-24T10:21:25.451332Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.451469Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-pcpj5 for: Get "http://localhost:37739/config_dump?include_eds=true": EOF 2025-06-24T10:21:25.454193Z error klog an error occurred forwarding 33007 -> 15020: error forwarding port 15020 to pod be4de9244cf6721d7393666e94817b13c69582dad2e421839dfac2d563ea6515, uid : port forward into network namespace "/var/run/netns/329b243b-3143-49db-80f5-e7a46341d615": failed to connect to localhost:15020 inside namespace be4de9244cf6721d7393666e94817b13c69582dad2e421839dfac2d563ea6515: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.454912Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-pcpj5 for: Get "http://localhost:37739/clusters": dial tcp 127.0.0.1:37739: connect: connection refused 2025-06-24T10:21:25.455011Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-h26fk for: Get "http://localhost:45509/stats/prometheus": dial tcp 127.0.0.1:45509: connect: connection refused 2025-06-24T10:21:25.455660Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.455692Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-qvvzr for: Get "http://localhost:33007/debug/ndsz": EOF 2025-06-24T10:21:25.456101Z error klog an error occurred forwarding 44567 -> 15020: error forwarding port 15020 to pod 6c9077d7c028cc7f7ea36db65b71ccef76c25231f60fa8cabfd9ee55cf29dff0, uid : port forward into network namespace "/var/run/netns/bc9cef9d-f0bd-47bf-bbe9-e3b6bacd553d": failed to connect to localhost:15020 inside namespace 6c9077d7c028cc7f7ea36db65b71ccef76c25231f60fa8cabfd9ee55cf29dff0: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.456327Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-qvvzr for: Get "http://localhost:33007/stats/prometheus": dial tcp 127.0.0.1:33007: connect: connection refused 2025-06-24T10:21:25.457709Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.457757Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-kp8w6 for: Get "http://localhost:44567/debug/ndsz": EOF 2025-06-24T10:21:25.458404Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-kp8w6 for: Get "http://localhost:44567/stats/prometheus": dial tcp 127.0.0.1:44567: connect: connection refused 2025-06-24T10:21:25.464375Z error klog an error occurred forwarding 40605 -> 15020: error forwarding port 15020 to pod c773a3c89fa565201eb599fc5696fe74808c2b9c913645c66d2230df3671b902, uid : port forward into network namespace "/var/run/netns/8585eab9-3e4b-4245-878c-64a4ba7fc8ad": failed to connect to localhost:15020 inside namespace c773a3c89fa565201eb599fc5696fe74808c2b9c913645c66d2230df3671b902: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.464431Z error klog an error occurred forwarding 42463 -> 15020: error forwarding port 15020 to pod 7b636f2f5405b6272599cb2c1c7fd4b84772492ecff6618acf769cc14ffe4e30, uid : port forward into network namespace "/var/run/netns/64cf4f09-0543-4672-9ab5-174741baeb44": failed to connect to localhost:15020 inside namespace 7b636f2f5405b6272599cb2c1c7fd4b84772492ecff6618acf769cc14ffe4e30: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:25.465912Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.465920Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb for: Get "http://localhost:42463/debug/ndsz": EOF 2025-06-24T10:21:25.465946Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-pcpj5 for: Get "http://localhost:40605/debug/ndsz": EOF 2025-06-24T10:21:25.465966Z error port forward failed: lost connection to pod 2025-06-24T10:21:25.466715Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-pcpj5 for: Get "http://localhost:40605/stats/prometheus": dial tcp 127.0.0.1:40605: connect: connection refused 2025-06-24T10:21:25.466729Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb for: Get "http://localhost:42463/stats/prometheus": dial tcp 127.0.0.1:42463: connect: connection refused 2025-06-24T10:21:25.485994Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:17Z/73d7ba6fd8-0ebf-4aa9-898e-603027e6ca55 2025-06-24T10:21:25.486118Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2532s 2025-06-24T10:21:25.486695Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:25.577396Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:17Z/73b8537c5b-75a1-49f0-ae7d-d09f7c83aade 2025-06-24T10:21:25.577728Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3447s 2025-06-24T10:21:25.578448Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:25.681253Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:17Z/7399177260-da18-444d-a030-4cb472e308a5 2025-06-24T10:21:25.681449Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4484s 2025-06-24T10:21:25.682377Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:26.505843Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-h26fk for: Get "http://localhost:46417/config_dump?include_eds=true": EOF 2025-06-24T10:21:26.536314Z error klog an error occurred forwarding 46417 -> 15000: error forwarding port 15000 to pod f6aeea49f1421af6353c8df4982128f912b25c486ca0fd78198c5e88e6b6c005, uid : port forward into network namespace "/var/run/netns/286b4106-157d-4c34-a87b-f9dffcac1d95": failed to connect to localhost:15000 inside namespace f6aeea49f1421af6353c8df4982128f912b25c486ca0fd78198c5e88e6b6c005: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:26.538250Z error port forward failed: lost connection to pod 2025-06-24T10:21:26.538280Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-h26fk for: Get "http://localhost:46417/clusters": EOF 2025-06-24T10:21:26.606296Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-qvvzr for: Get "http://localhost:40221/config_dump?include_eds=true": EOF 2025-06-24T10:21:26.616758Z error klog an error occurred forwarding 40221 -> 15000: error forwarding port 15000 to pod be4de9244cf6721d7393666e94817b13c69582dad2e421839dfac2d563ea6515, uid : sandbox be4de9244cf6721d7393666e94817b13c69582dad2e421839dfac2d563ea6515 is not running: Unhandled Error logger=UnhandledError 2025-06-24T10:21:26.618897Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-qvvzr for: Get "http://localhost:40221/clusters": EOF 2025-06-24T10:21:26.618942Z error port forward failed: lost connection to pod 2025-06-24T10:21:26.728746Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb for: Get "http://localhost:34799/config_dump?include_eds=true": EOF 2025-06-24T10:21:26.734312Z error klog an error occurred forwarding 34799 -> 15000: error forwarding port 15000 to pod 7b636f2f5405b6272599cb2c1c7fd4b84772492ecff6618acf769cc14ffe4e30, uid : port forward into network namespace "/var/run/netns/64cf4f09-0543-4672-9ab5-174741baeb44": failed to connect to localhost:15000 inside namespace 7b636f2f5405b6272599cb2c1c7fd4b84772492ecff6618acf769cc14ffe4e30: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:21:26.736143Z error port forward failed: lost connection to pod 2025-06-24T10:21:26.736157Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-5tzsb for: Get "http://localhost:34799/clusters": EOF 2025-06-24T10:21:26.747916Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:21:26.747950Z info tf === SUCCEEDED: Cleanup Istio in 23.967µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root DONE 2 runs, 31 tests, 18 failures in 505.461s exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0/server:server-naked-foo\[success\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3432921 2025-06-24T10:21:37.109221Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:21:37.109262Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:21:37.139521Z info tf === Test Framework Settings === 2025-06-24T10:21:37.139611Z info tf TestID: security_ca_custom_root RunID: eba3c97f-f8da-46bc-9f37-c07d1a099e28 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:21:37.139619Z info tf =============================== 2025-06-24T10:21:37.139779Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97 2025-06-24T10:21:37.139865Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:21:37.139878Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:21:37.139896Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:21:37.139905Z info tf === BEGIN: Building clusters === 2025-06-24T10:21:37.142749Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:21:37.142758Z info tf === DONE: Building clusters === 2025-06-24T10:21:37.142774Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:21:37.181331Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:21:37.181376Z info tf === SUCCESS: context setup function === 2025-06-24T10:21:37.181384Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:21:37.181399Z info tf === Istio Component Config === 2025-06-24T10:21:37.181455Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:21:37.181466Z info tf ================================ 2025-06-24T10:21:37.184412Z info tf skipping deployment as specified in the config 2025-06-24T10:21:37.184429Z info tf === SUCCEEDED: Deploy Istio in 3.041761ms [Suite=security_ca_custom_root]=== 2025-06-24T10:21:37.184438Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:21:37.257199Z info tf === SUCCEEDED: Create namespace echo1 in 72.755102ms === Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************************************************************************************************************************************************+++++ *******************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97/_suite_context/ca-custom-root-1779211994/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *****************************************************************************************************************************************************************************************************************+++++ ************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97/_suite_context/ca-custom-root-1779211994/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***********************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97/_suite_context/ca-custom-root-1779211994/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:21:38.665410Z info tf === BEGIN: Create namespace external === 2025-06-24T10:21:38.693541Z info tf === SUCCEEDED: Create namespace external in 28.137851ms === 2025-06-24T10:21:38.728461Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.732521Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.764062Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.774101Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.781722Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.797766Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.801472Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.806418Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.810306Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.818864Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.821527Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.825738Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.828880Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.859427Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.864537Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.869310Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.871841Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.877691Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.880601Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.887236Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.890223Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.954226Z warn tf failed getting labels for namespace echo1, assuming ambient is on 2025-06-24T10:21:38.960377Z warn tf failed getting labels for namespace echo1, assuming injection is on 2025-06-24T10:21:38.963850Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:21:39.037823Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:21:39.037894Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97/_suite_context/ServiceAccount-Service.211954341.yaml] to ns echo1 in cluster cluster-0: apply: patch: namespaces "echo1" not found 2025-06-24T10:21:39.037946Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:39.037946Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:39.037987Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:39.105814Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97/_suite_context/ServiceAccount-Service.211954341.yaml] to ns echo1 in cluster cluster-0: apply: patch: namespaces "echo1" not found 2025-06-24T10:21:39.105857Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97/_suite_context/ServiceAccount-Service.211954341.yaml] to ns echo1 in cluster cluster-0: apply: patch: namespaces "echo1" not found) === 2025-06-24T10:21:39.105864Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eba3c97/_suite_context/ServiceAccount-Service.211954341.yaml] to ns echo1 in cluster cluster-0: apply: patch: namespaces "echo1" not found 2025-06-24T10:21:39.105887Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:39.105923Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:39.105899Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:21:39.235435Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:21:39Z/75d3af7cc3-a289-4115-8d39-341ef14aa46f 2025-06-24T10:21:39.235658Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1973s 2025-06-24T10:21:39.236295Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:39.251063Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:21:39.251127Z info tf === SUCCEEDED: Cleanup Istio in 60.148µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3434288 2025-06-24T10:21:50.197213Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:21:50.197272Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:21:50.226573Z info tf === Test Framework Settings === 2025-06-24T10:21:50.226659Z info tf TestID: security_ca_custom_root RunID: 22c41945-4a79-4633-85d5-842a776d060b NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:21:50.226664Z info tf =============================== 2025-06-24T10:21:50.226827Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-22c4194 2025-06-24T10:21:50.226900Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:21:50.226909Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:21:50.226926Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:21:50.226932Z info tf === BEGIN: Building clusters === 2025-06-24T10:21:50.232093Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:21:50.232179Z info tf === DONE: Building clusters === 2025-06-24T10:21:50.232202Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:21:50.265021Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:21:50.265085Z info tf === SUCCESS: context setup function === 2025-06-24T10:21:50.265099Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:21:50.265127Z info tf === Istio Component Config === 2025-06-24T10:21:50.265221Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:21:50.265232Z info tf ================================ 2025-06-24T10:21:50.269631Z info tf skipping deployment as specified in the config 2025-06-24T10:21:50.269656Z info tf === SUCCEEDED: Deploy Istio in 4.55432ms [Suite=security_ca_custom_root]=== 2025-06-24T10:21:50.269669Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:21:50.353807Z info tf === SUCCEEDED: Create namespace echo1 in 84.134952ms === Generating RSA private key, 2048 bit long modulus (2 primes) **************+++++ ******************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-22c4194/_suite_context/ca-custom-root-728995859/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-22c4194/_suite_context/ca-custom-root-728995859/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-22c4194/_suite_context/ca-custom-root-728995859/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:21:51.545066Z info tf === BEGIN: Create namespace external === 2025-06-24T10:21:51.592126Z info tf === SUCCEEDED: Create namespace external in 47.075505ms === 2025-06-24T10:21:51.755366Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:21:52.972514Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:52.973071Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:52.995120Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.002927Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.041816Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.051382Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.055804Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.055982Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.056032Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.060122Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.065026Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.072784Z info klog Caches are synced for echo 2025-06-24T10:21:53.074154Z info klog Caches are synced for echo 2025-06-24T10:21:53.081473Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.095658Z info klog Caches are synced for echo 2025-06-24T10:21:53.098860Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.103007Z info klog Caches are synced for echo 2025-06-24T10:21:53.105010Z info klog Waiting for caches to sync for echo 2025-06-24T10:21:53.142588Z info klog Caches are synced for echo 2025-06-24T10:21:53.152280Z info klog Caches are synced for echo 2025-06-24T10:21:53.156763Z info klog Caches are synced for echo 2025-06-24T10:21:53.156803Z info klog Caches are synced for echo 2025-06-24T10:21:53.156820Z info klog Caches are synced for echo 2025-06-24T10:21:53.162403Z info klog Caches are synced for echo 2025-06-24T10:21:53.166100Z info klog Caches are synced for echo 2025-06-24T10:21:53.182262Z info klog Caches are synced for echo 2025-06-24T10:21:53.199799Z info klog Caches are synced for echo 2025-06-24T10:21:53.205747Z info klog Caches are synced for echo 2025-06-24T10:22:12.293412Z info tf === SUCCEEDED: Deploy echo instances in 20.538041262s === 2025-06-24T10:22:12.328735Z info tf === DONE: Setup: 'security_ca_custom_root' (22.096519062s) === 2025-06-24T10:22:12.328802Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] 2025-06-24T10:22:12.346184Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.3.67:41246->172.30.112.136:443: read: connection reset by peer 2025-06-24T10:22:42.353050Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006863046s)' === 2025-06-24T10:22:42.353100Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:22:42.353094Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:22:42.353116Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:22:43.448918Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86af6823cc-f3bf-4732-b748-17730691dbe0 2025-06-24T10:22:43.449168Z info tf istioctl ([x internal-debug --all configz]): completed after 1.0957s 2025-06-24T10:22:43.449724Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:43.539562Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86bad295bb-b242-4abc-bc3a-4577b1f4a535 2025-06-24T10:22:43.539726Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1862s 2025-06-24T10:22:43.540254Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:43.630966Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/865e41ee7a-f616-44f0-8db6-9f105715b1c2 2025-06-24T10:22:43.631114Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.2776s 2025-06-24T10:22:43.631650Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (31.99s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 2) (31.99s) PASS tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-bar[fail] (re-run 2) (0.03s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0 2025-06-24T10:22:12.345991Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-06-24T10:22:44.364746Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (32.018754562s)' === 2025-06-24T10:22:44.364796Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:22:44.364805Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:22:44.364835Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:22:45.001870Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/860b15012e-bf60-414b-b2fa-111d8bab308f 2025-06-24T10:22:45.002120Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6370s 2025-06-24T10:22:45.002734Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:45.203131Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86892a7d20-2d11-435e-a231-d0aaec034636 2025-06-24T10:22:45.203262Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8381s 2025-06-24T10:22:45.203864Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:45.293394Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/869b713234-87ba-43e9-bed1-e6705cd19910 2025-06-24T10:22:45.293563Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9283s 2025-06-24T10:22:45.294090Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0 (33.54s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 2) (33.54s) === RUN TestTrustDomainAliasSecureNaming 2025-06-24T10:22:12.329300Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-06-24T10:22:45.881530Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (33.552224275s)' === 2025-06-24T10:22:45.881599Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:22:45.881612Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:22:45.881617Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:22:46.111697Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86c40cbc5e-7c67-4ec5-be81-c60644350851 2025-06-24T10:22:46.111953Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2302s 2025-06-24T10:22:46.112421Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:46.225778Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86a72730d3-3f97-414e-92e7-c86666ea27aa 2025-06-24T10:22:46.225883Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3441s 2025-06-24T10:22:46.226365Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:46.325864Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/8605827fe3-00a6-436b-b4d8-ef7699d6e450 2025-06-24T10:22:46.325938Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4441s 2025-06-24T10:22:46.326556Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming (34.62s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming (re-run 2) (34.62s) 2025-06-24T10:22:46.950160Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-06-24T10:22:46.950223Z info tf === Suite "security_ca_custom_root" run time: 56.71802685s === 2025-06-24T10:22:46.950243Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-06-24T10:22:46.950721Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:22:46.950742Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:22:46.950704Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:22:47.210245Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:47Z/878a63179b-3336-41f3-b65f-3ad565407ce3 2025-06-24T10:22:47.210449Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2593s 2025-06-24T10:22:47.211205Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:47.346774Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:47Z/870eff11c1-206b-4183-b9da-7ef87b9a8481 2025-06-24T10:22:47.347052Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3959s 2025-06-24T10:22:47.347950Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:47.428442Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:47Z/87d53b56dd-c207-4b26-b8d1-fb8c69903819 2025-06-24T10:22:47.428604Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4774s 2025-06-24T10:22:47.429249Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:54.133383Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:22:54.133506Z info tf === SUCCEEDED: Cleanup Istio in 112.134µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:http-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3435805 2025-06-24T10:23:05.552355Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:23:05.552404Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:23:05.581442Z info tf === Test Framework Settings === 2025-06-24T10:23:05.581536Z info tf TestID: security_ca_custom_root RunID: 78093fd2-f3cb-4134-be3a-6adddc0fd09d NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:23:05.581544Z info tf =============================== 2025-06-24T10:23:05.581807Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd 2025-06-24T10:23:05.581931Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:23:05.581945Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:23:05.581966Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:23:05.581975Z info tf === BEGIN: Building clusters === 2025-06-24T10:23:05.584363Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:23:05.584370Z info tf === DONE: Building clusters === 2025-06-24T10:23:05.584390Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:23:05.656083Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:23:05.656199Z info tf === SUCCESS: context setup function === 2025-06-24T10:23:05.656211Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:23:05.656237Z info tf === Istio Component Config === 2025-06-24T10:23:05.656326Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:23:05.656335Z info tf ================================ 2025-06-24T10:23:05.660711Z info tf skipping deployment as specified in the config 2025-06-24T10:23:05.660734Z info tf === SUCCEEDED: Deploy Istio in 4.518512ms [Suite=security_ca_custom_root]=== 2025-06-24T10:23:05.660744Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:23:05.706967Z info tf === SUCCEEDED: Create namespace echo1 in 46.205351ms === Generating RSA private key, 2048 bit long modulus (2 primes) ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ **************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd/_suite_context/ca-custom-root-1656695757/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************************************************+++++ ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd/_suite_context/ca-custom-root-1656695757/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *************************************************************************************************************************************************************************+++++ **********************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd/_suite_context/ca-custom-root-1656695757/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:23:07.418681Z info tf === BEGIN: Create namespace external === 2025-06-24T10:23:07.426810Z info tf === SUCCEEDED: Create namespace external in 8.152543ms === 2025-06-24T10:23:07.505141Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:23:07.525437Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:23:07.525474Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd/_suite_context/Service.441588064.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:23:07.525508Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:23:07.525515Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:23:07.525591Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:23:07.598907Z error klog an error occurred forwarding 38701 -> 15020: error forwarding port 15020 to pod 516a43004f44ee80c451b1b182ee2c31e788f778ddb0dcdb54023497ee0422e1, uid : port forward into network namespace "/var/run/netns/2f2cad2f-88ee-47c8-bb62-779011d4f405": failed to connect to localhost:15020 inside namespace 516a43004f44ee80c451b1b182ee2c31e788f778ddb0dcdb54023497ee0422e1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.602158Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-w4r7t for: Get "http://localhost:38701/debug/ndsz": EOF 2025-06-24T10:23:07.602557Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.603589Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-w4r7t for: Get "http://localhost:38701/stats/prometheus": dial tcp 127.0.0.1:38701: connect: connection refused 2025-06-24T10:23:07.647526Z error klog an error occurred forwarding 43147 -> 15020: error forwarding port 15020 to pod 43a4bebbf7b0c8003ae3eaffde1e37ffcdd900c4507c671ca5a41584c6d01631, uid : port forward into network namespace "/var/run/netns/c2fdd030-b7a1-482c-911e-5d01370bb4ba": failed to connect to localhost:15020 inside namespace 43a4bebbf7b0c8003ae3eaffde1e37ffcdd900c4507c671ca5a41584c6d01631: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.649189Z error klog an error occurred forwarding 33163 -> 15020: error forwarding port 15020 to pod 290efdfe374a5ff6068b0b0ace39fda6728585d233749073bf2646e1ddae3f65, uid : port forward into network namespace "/var/run/netns/1e39431b-8780-4faa-9790-866f48a23cb4": failed to connect to localhost:15020 inside namespace 290efdfe374a5ff6068b0b0ace39fda6728585d233749073bf2646e1ddae3f65: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.649635Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.649636Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-b5bs4 for: Get "http://localhost:43147/debug/ndsz": EOF 2025-06-24T10:23:07.650494Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-b5bs4 for: Get "http://localhost:43147/stats/prometheus": dial tcp 127.0.0.1:43147: connect: connection refused 2025-06-24T10:23:07.650780Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-wmzpb for: Get "http://localhost:33163/debug/ndsz": EOF 2025-06-24T10:23:07.650809Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.651316Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-wmzpb for: Get "http://localhost:33163/stats/prometheus": dial tcp 127.0.0.1:33163: connect: connection refused 2025-06-24T10:23:07.668195Z error klog an error occurred forwarding 45563 -> 15020: error forwarding port 15020 to pod ecb8a672bb73ff522f64e0b34d3ce88d94f5777fc0d4d1f11e0f8c92653074fc, uid : port forward into network namespace "/var/run/netns/55fb55d9-d6db-44c5-9fdb-ff7dd0cf4257": failed to connect to localhost:15020 inside namespace ecb8a672bb73ff522f64e0b34d3ce88d94f5777fc0d4d1f11e0f8c92653074fc: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.671396Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-gl2sd for: Get "http://localhost:45563/debug/ndsz": EOF 2025-06-24T10:23:07.671487Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.673590Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-gl2sd for: Get "http://localhost:45563/stats/prometheus": dial tcp 127.0.0.1:45563: connect: connection refused 2025-06-24T10:23:07.688186Z error klog an error occurred forwarding 40199 -> 15020: error forwarding port 15020 to pod 9dccb25a952c41dbab10e45419ce850fbfd5d334243ee1909c8fd51e4e0bc3c8, uid : port forward into network namespace "/var/run/netns/9448dac6-2caf-4033-82c7-63bab4a24eba": failed to connect to localhost:15020 inside namespace 9dccb25a952c41dbab10e45419ce850fbfd5d334243ee1909c8fd51e4e0bc3c8: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.690164Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:40199/debug/ndsz": EOF 2025-06-24T10:23:07.690212Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.691474Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:40199/stats/prometheus": dial tcp 127.0.0.1:40199: connect: connection refused 2025-06-24T10:23:07.705987Z error klog an error occurred forwarding 42735 -> 15020: error forwarding port 15020 to pod fcb4a92f03fd33aa8f31acf870ddcdc2b8f6ccb521eeb3cc854fa8a7e56b3c1b, uid : port forward into network namespace "/var/run/netns/36b826fd-1a7a-4a53-a721-3f9842cc5f97": failed to connect to localhost:15020 inside namespace fcb4a92f03fd33aa8f31acf870ddcdc2b8f6ccb521eeb3cc854fa8a7e56b3c1b: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.707783Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-gpxzx for: Get "http://localhost:42735/debug/ndsz": EOF 2025-06-24T10:23:07.707814Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.708476Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-gpxzx for: Get "http://localhost:42735/stats/prometheus": dial tcp 127.0.0.1:42735: connect: connection refused 2025-06-24T10:23:07.734253Z error klog an error occurred forwarding 44673 -> 15020: error forwarding port 15020 to pod d812735de59e65242fd388544880b05e71099a1d0e8462351ca5eb5692bb2aad, uid : port forward into network namespace "/var/run/netns/bcffe39a-4d49-4820-a46b-6da3e56412e7": failed to connect to localhost:15020 inside namespace d812735de59e65242fd388544880b05e71099a1d0e8462351ca5eb5692bb2aad: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.736521Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.736563Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-r54jz for: Get "http://localhost:44673/debug/ndsz": EOF 2025-06-24T10:23:07.751616Z error klog an error occurred forwarding 34057 -> 15020: error forwarding port 15020 to pod cc6efdc778b810e395dc3b4ef73ecf82f9383e4ffcb5df6b7029b2f87fbde4af, uid : port forward into network namespace "/var/run/netns/51d8a828-cf91-4402-a98c-69c1ac041c25": failed to connect to localhost:15020 inside namespace cc6efdc778b810e395dc3b4ef73ecf82f9383e4ffcb5df6b7029b2f87fbde4af: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:07.751834Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:07Z/93722e6de5-cf71-4e01-bc8c-bd62a8618534 2025-06-24T10:23:07.752136Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2264s 2025-06-24T10:23:07.752742Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:23:07.753219Z error port forward failed: lost connection to pod 2025-06-24T10:23:07.753246Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-6qtjl for: Get "http://localhost:34057/debug/ndsz": EOF 2025-06-24T10:23:07.852700Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-r54jz for: Get "http://localhost:44673/stats/prometheus": dial tcp 127.0.0.1:44673: connect: connection refused 2025-06-24T10:23:07.852730Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-6qtjl for: Get "http://localhost:34057/stats/prometheus": dial tcp 127.0.0.1:34057: connect: connection refused 2025-06-24T10:23:07.918769Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:07Z/93629abe29-6bc9-411b-bed7-656529c7d3e5 2025-06-24T10:23:07.918943Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3932s 2025-06-24T10:23:07.919655Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:23:08.029351Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:07Z/93ed175735-dae8-4232-94f6-b5cdf2d6c5e7 2025-06-24T10:23:08.029591Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5038s 2025-06-24T10:23:08.030587Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:23:08.700530Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd/_suite_context/Service.441588064.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:23:08.700599Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd/_suite_context/Service.441588064.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-06-24T10:23:08.700606Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-78093fd/_suite_context/Service.441588064.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:23:08.700638Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:23:08.700665Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:23:08.700678Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:23:08.832736Z error klog an error occurred forwarding 37599 -> 15020: error forwarding port 15020 to pod 290efdfe374a5ff6068b0b0ace39fda6728585d233749073bf2646e1ddae3f65, uid : port forward into network namespace "/var/run/netns/1e39431b-8780-4faa-9790-866f48a23cb4": failed to connect to localhost:15020 inside namespace 290efdfe374a5ff6068b0b0ace39fda6728585d233749073bf2646e1ddae3f65: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.833214Z error klog an error occurred forwarding 38033 -> 15020: error forwarding port 15020 to pod 516a43004f44ee80c451b1b182ee2c31e788f778ddb0dcdb54023497ee0422e1, uid : port forward into network namespace "/var/run/netns/2f2cad2f-88ee-47c8-bb62-779011d4f405": failed to connect to localhost:15020 inside namespace 516a43004f44ee80c451b1b182ee2c31e788f778ddb0dcdb54023497ee0422e1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.833402Z error klog an error occurred forwarding 34503 -> 15020: error forwarding port 15020 to pod cc6efdc778b810e395dc3b4ef73ecf82f9383e4ffcb5df6b7029b2f87fbde4af, uid : port forward into network namespace "/var/run/netns/51d8a828-cf91-4402-a98c-69c1ac041c25": failed to connect to localhost:15020 inside namespace cc6efdc778b810e395dc3b4ef73ecf82f9383e4ffcb5df6b7029b2f87fbde4af: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.833490Z error klog an error occurred forwarding 45263 -> 15020: error forwarding port 15020 to pod fcb4a92f03fd33aa8f31acf870ddcdc2b8f6ccb521eeb3cc854fa8a7e56b3c1b, uid : port forward into network namespace "/var/run/netns/36b826fd-1a7a-4a53-a721-3f9842cc5f97": failed to connect to localhost:15020 inside namespace fcb4a92f03fd33aa8f31acf870ddcdc2b8f6ccb521eeb3cc854fa8a7e56b3c1b: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.834373Z error klog an error occurred forwarding 34089 -> 15020: error forwarding port 15020 to pod d812735de59e65242fd388544880b05e71099a1d0e8462351ca5eb5692bb2aad, uid : port forward into network namespace "/var/run/netns/bcffe39a-4d49-4820-a46b-6da3e56412e7": failed to connect to localhost:15020 inside namespace d812735de59e65242fd388544880b05e71099a1d0e8462351ca5eb5692bb2aad: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.834801Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.834825Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-6qtjl for: Get "http://localhost:34503/debug/ndsz": EOF 2025-06-24T10:23:08.834809Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.834832Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-wmzpb for: Get "http://localhost:37599/debug/ndsz": EOF 2025-06-24T10:23:08.834967Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-w4r7t for: Get "http://localhost:38033/debug/ndsz": EOF 2025-06-24T10:23:08.834990Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.835358Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.835333Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-gpxzx for: Get "http://localhost:45263/debug/ndsz": EOF 2025-06-24T10:23:08.835491Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-78f49b749b-w4r7t for: Get "http://localhost:38033/stats/prometheus": dial tcp 127.0.0.1:38033: connect: connection refused 2025-06-24T10:23:08.835572Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-546c55dc9c-wmzpb for: Get "http://localhost:37599/stats/prometheus": dial tcp 127.0.0.1:37599: connect: connection refused 2025-06-24T10:23:08.835674Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-65f898ccd5-6qtjl for: Get "http://localhost:34503/stats/prometheus": dial tcp 127.0.0.1:34503: connect: connection refused 2025-06-24T10:23:08.835875Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-r54jz for: Get "http://localhost:34089/debug/ndsz": EOF 2025-06-24T10:23:08.836015Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.836053Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-7bcb76fc6f-gpxzx for: Get "http://localhost:45263/stats/prometheus": dial tcp 127.0.0.1:45263: connect: connection refused 2025-06-24T10:23:08.836295Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-86cfb5b889-r54jz for: Get "http://localhost:34089/stats/prometheus": dial tcp 127.0.0.1:34089: connect: connection refused 2025-06-24T10:23:08.840763Z error klog an error occurred forwarding 40077 -> 15020: error forwarding port 15020 to pod ecb8a672bb73ff522f64e0b34d3ce88d94f5777fc0d4d1f11e0f8c92653074fc, uid : port forward into network namespace "/var/run/netns/55fb55d9-d6db-44c5-9fdb-ff7dd0cf4257": failed to connect to localhost:15020 inside namespace ecb8a672bb73ff522f64e0b34d3ce88d94f5777fc0d4d1f11e0f8c92653074fc: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.842064Z error klog an error occurred forwarding 36935 -> 15020: error forwarding port 15020 to pod 9dccb25a952c41dbab10e45419ce850fbfd5d334243ee1909c8fd51e4e0bc3c8, uid : port forward into network namespace "/var/run/netns/9448dac6-2caf-4033-82c7-63bab4a24eba": failed to connect to localhost:15020 inside namespace 9dccb25a952c41dbab10e45419ce850fbfd5d334243ee1909c8fd51e4e0bc3c8: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.842639Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-gl2sd for: Get "http://localhost:40077/debug/ndsz": EOF 2025-06-24T10:23:08.842642Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.843286Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-64d65d6c8b-gl2sd for: Get "http://localhost:40077/stats/prometheus": dial tcp 127.0.0.1:40077: connect: connection refused 2025-06-24T10:23:08.843667Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.843690Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:36935/debug/ndsz": EOF 2025-06-24T10:23:08.844299Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:36935/stats/prometheus": dial tcp 127.0.0.1:36935: connect: connection refused 2025-06-24T10:23:08.872017Z error klog an error occurred forwarding 38979 -> 15020: error forwarding port 15020 to pod 43a4bebbf7b0c8003ae3eaffde1e37ffcdd900c4507c671ca5a41584c6d01631, uid : port forward into network namespace "/var/run/netns/c2fdd030-b7a1-482c-911e-5d01370bb4ba": failed to connect to localhost:15020 inside namespace 43a4bebbf7b0c8003ae3eaffde1e37ffcdd900c4507c671ca5a41584c6d01631: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:23:08.874285Z error port forward failed: lost connection to pod 2025-06-24T10:23:08.874288Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-b5bs4 for: Get "http://localhost:38979/debug/ndsz": EOF 2025-06-24T10:23:08.875201Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-6975698f5d-b5bs4 for: Get "http://localhost:38979/stats/prometheus": dial tcp 127.0.0.1:38979: connect: connection refused 2025-06-24T10:23:08.881477Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:07Z/93615f880c-1cca-429e-a3da-eb1d98427173 2025-06-24T10:23:08.881636Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1807s 2025-06-24T10:23:08.882222Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:23:09.047542Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:07Z/93bf40411d-b84c-4c13-9140-3f6d1a6fd6b0 2025-06-24T10:23:09.047732Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3467s 2025-06-24T10:23:09.048317Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:23:09.249431Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:07Z/93520a4670-5c0a-4f20-bb30-6ee5b9ad8ec5 2025-06-24T10:23:09.249611Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5486s 2025-06-24T10:23:09.250434Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:23:09.686015Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:23:09.686095Z info tf === SUCCEEDED: Cleanup Istio in 75.063µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3438769 2025-06-24T10:23:20.252688Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:23:20.252778Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:23:20.291608Z info tf === Test Framework Settings === 2025-06-24T10:23:20.291740Z info tf TestID: security_ca_custom_root RunID: b1022d97-60fc-4820-8336-96fbb14d6db2 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:23:20.291750Z info tf =============================== 2025-06-24T10:23:20.291963Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-b1022d9 2025-06-24T10:23:20.292085Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:23:20.292107Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:23:20.292132Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:23:20.292145Z info tf === BEGIN: Building clusters === 2025-06-24T10:23:20.294681Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:23:20.294690Z info tf === DONE: Building clusters === 2025-06-24T10:23:20.294709Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:23:20.335259Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:23:20.335358Z info tf === SUCCESS: context setup function === 2025-06-24T10:23:20.335368Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:23:20.335386Z info tf === Istio Component Config === 2025-06-24T10:23:20.335455Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:23:20.335463Z info tf ================================ 2025-06-24T10:23:20.340335Z info tf skipping deployment as specified in the config 2025-06-24T10:23:20.340358Z info tf === SUCCEEDED: Deploy Istio in 4.98618ms [Suite=security_ca_custom_root]=== 2025-06-24T10:23:20.340370Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:23:20.419208Z info tf === SUCCEEDED: Create namespace echo1 in 78.823108ms === Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-b1022d9/_suite_context/ca-custom-root-3685766628/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ************************************************************************************************************************************************************+++++ *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-b1022d9/_suite_context/ca-custom-root-3685766628/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ************************************************************************************************************************************************************+++++ *************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-b1022d9/_suite_context/ca-custom-root-3685766628/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:23:22.556761Z info tf === BEGIN: Create namespace external === 2025-06-24T10:23:22.582620Z info tf === SUCCEEDED: Create namespace external in 25.886257ms === 2025-06-24T10:23:22.761257Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:23:24.497399Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.500995Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.518804Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.519179Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.543897Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.556997Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.560455Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.590321Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.590413Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.594246Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.597821Z info klog Caches are synced for echo 2025-06-24T10:23:24.601901Z info klog Caches are synced for echo 2025-06-24T10:23:24.602761Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.619787Z info klog Caches are synced for echo 2025-06-24T10:23:24.619895Z info klog Caches are synced for echo 2025-06-24T10:23:24.632615Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.644700Z info klog Caches are synced for echo 2025-06-24T10:23:24.647915Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.653426Z info klog Waiting for caches to sync for echo 2025-06-24T10:23:24.658164Z info klog Caches are synced for echo 2025-06-24T10:23:24.660661Z info klog Caches are synced for echo 2025-06-24T10:23:24.691180Z info klog Caches are synced for echo 2025-06-24T10:23:24.691258Z info klog Caches are synced for echo 2025-06-24T10:23:24.694801Z info klog Caches are synced for echo 2025-06-24T10:23:24.703252Z info klog Caches are synced for echo 2025-06-24T10:23:24.733290Z info klog Caches are synced for echo 2025-06-24T10:23:24.748062Z info klog Caches are synced for echo 2025-06-24T10:23:24.754615Z info klog Caches are synced for echo 2025-06-24T10:23:43.843543Z info tf === SUCCEEDED: Deploy echo instances in 21.082279818s === 2025-06-24T10:23:43.874899Z info tf === DONE: Setup: 'security_ca_custom_root' (23.580175922s) === 2025-06-24T10:23:43.874936Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:http-plaintext[allow] (re-run 2) (0.06s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:tcp-plaintext[allow] (re-run 2) (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:http-plaintext[allow] (re-run 2) (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:tcp-plaintext[allow] (re-run 2) (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:http-mtls[allow] (re-run 2) (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:tcp-mtls[allow] (re-run 2) (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:http-mtls[deny] (re-run 2) (0.71s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls[deny] (re-run 2) (0.05s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls-pass-through[deny] (re-run 2) (0.04s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-06-24T10:23:44.944909Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:24:14.946250Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.001336344s)' === 2025-06-24T10:24:14.946332Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:24:14.946330Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:24:14.946362Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:24:15.226949Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1011beb4930-5e73-43c8-90c5-0bc85f9aa040 2025-06-24T10:24:15.227328Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2807s 2025-06-24T10:24:15.227864Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:15.405526Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1019f6f97a0-dba1-436d-86d5-2229c058c0ac 2025-06-24T10:24:15.405822Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4591s 2025-06-24T10:24:15.406912Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:15.516013Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10123f09d91-3788-4948-bca7-f97c217b050d 2025-06-24T10:24:15.516231Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5695s 2025-06-24T10:24:15.516677Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (31.38s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 2) (31.38s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] 2025-06-24T10:24:16.326302Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-06-24T10:24:35.180024Z error port forward failed: lost connection to pod 2025-06-24T10:24:35.603733Z warn pod: external/external-v1-f9d74566f-59fvt port forward terminated 2025-06-24T10:24:35.684012Z warn pod: external/external-v1-f9d74566f-59fvt port forward reconnect success 2025-06-24T10:24:36.003721Z error port forward failed: lost connection to pod 2025-06-24T10:24:36.204263Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-pm52m port forward terminated 2025-06-24T10:24:36.236700Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-pm52m port forward reconnect success 2025-06-24T10:24:36.256334Z error port forward failed: lost connection to pod 2025-06-24T10:24:36.258577Z error port forward failed: lost connection to pod 2025-06-24T10:24:36.725472Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-jg9qk port forward terminated 2025-06-24T10:24:36.725423Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-zl8xx port forward terminated 2025-06-24T10:24:36.760903Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-zl8xx port forward reconnect success 2025-06-24T10:24:36.763549Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-jg9qk port forward reconnect success 2025-06-24T10:24:39.207463Z error port forward failed: lost connection to pod 2025-06-24T10:24:39.428318Z warn pod: echo1/proxyless-grpc-v1-8996df688-kk77f port forward terminated 2025-06-24T10:24:39.465065Z warn pod: echo1/proxyless-grpc-v1-8996df688-kk77f port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:24:46.327056Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.000751017s)' === 2025-06-24T10:24:46.327134Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:24:46.327128Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:24:46.327201Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:24:46.659123Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1019c01650a-51f5-47ee-880f-a51069feb1da 2025-06-24T10:24:46.659385Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3317s 2025-06-24T10:24:46.660035Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:46.813668Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101fbb18efc-d664-45ed-89c3-896dd8c6c77a 2025-06-24T10:24:46.813792Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4861s 2025-06-24T10:24:46.814523Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:46.949459Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101ad90b4d2-1a1a-48ef-9580-0d80c4bdd503 2025-06-24T10:24:46.949615Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6219s 2025-06-24T10:24:46.950497Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (31.03s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (re-run 2) (31.03s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] 2025-06-24T10:24:47.356232Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-06-24T10:24:47.496517Z error port forward failed: lost connection to pod 2025-06-24T10:24:47.819520Z error port forward failed: lost connection to pod 2025-06-24T10:24:47.959899Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:24:47.994231Z error port forward failed: lost connection to pod 2025-06-24T10:24:47.994511Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:24:48.223193Z warn pod: echo1/sotw-v1-65f898ccd5-pt4z4 port forward terminated 2025-06-24T10:24:48.262151Z warn pod: echo1/sotw-v1-65f898ccd5-pt4z4 port forward reconnect success 2025-06-24T10:24:48.433702Z warn pod: echo1/b-v1-7bcb76fc6f-29ngt port forward terminated 2025-06-24T10:24:48.465127Z warn pod: echo1/b-v1-7bcb76fc6f-29ngt port forward reconnect success 2025-06-24T10:24:48.616263Z error port forward failed: lost connection to pod 2025-06-24T10:24:48.815694Z error port forward failed: lost connection to pod 2025-06-24T10:24:48.977188Z warn pod: echo1/server-v1-86cfb5b889-csx2d port forward terminated 2025-06-24T10:24:49.017709Z warn pod: echo1/server-v1-86cfb5b889-csx2d port forward reconnect success 2025-06-24T10:24:49.134306Z error port forward failed: lost connection to pod 2025-06-24T10:24:49.279037Z warn pod: echo1/a-v1-78f49b749b-trkfh port forward terminated 2025-06-24T10:24:49.305443Z warn pod: echo1/headless-v1-64d65d6c8b-t549v port forward terminated 2025-06-24T10:24:49.311197Z warn pod: echo1/a-v1-78f49b749b-trkfh port forward reconnect success 2025-06-24T10:24:49.336304Z warn pod: echo1/headless-v1-64d65d6c8b-t549v port forward reconnect success 2025-06-24T10:24:49.562596Z error port forward failed: lost connection to pod 2025-06-24T10:24:50.028499Z warn pod: echo1/c-v1-6975698f5d-gp279 port forward terminated 2025-06-24T10:24:50.067687Z warn pod: echo1/c-v1-6975698f5d-gp279 port forward reconnect success 2025-06-24T10:24:50.154509Z error port forward failed: lost connection to pod 2025-06-24T10:24:50.319988Z warn pod: echo1/client-v1-546c55dc9c-8ddkj port forward terminated 2025-06-24T10:24:50.354763Z warn pod: echo1/client-v1-546c55dc9c-8ddkj port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.72:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.72:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:25:17.357221Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000982736s)' === 2025-06-24T10:25:17.357336Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:25:17.357396Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:25:17.357362Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:25:17.581705Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10128d64634-ba83-437a-98af-3cb33d1ac4e6 2025-06-24T10:25:17.581960Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2242s 2025-06-24T10:25:17.582698Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:17.680443Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1011ed2e4f7-6b67-4c9b-9778-cff8c8e3972e 2025-06-24T10:25:17.680731Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3229s 2025-06-24T10:25:17.681622Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:17.796578Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101178ab1c4-a9ec-4375-a027-21b96ef319e9 2025-06-24T10:25:17.796783Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4390s 2025-06-24T10:25:17.797557Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (30.94s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 2) (30.94s) === RUN TestTrustDomainValidation/From_primary-0 2025-06-24T10:23:43.898553Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:25:18.295056Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m34.396495187s)' === 2025-06-24T10:25:18.295171Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:25:18.295171Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:25:18.295183Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:25:18.463440Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101a0c57c52-f941-40e8-88a7-0378cd89bc6b 2025-06-24T10:25:18.463636Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1682s 2025-06-24T10:25:18.464375Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:18.588282Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10136856fc4-8f7e-4c88-90f9-4ce62dc758a7 2025-06-24T10:25:18.588623Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2932s 2025-06-24T10:25:18.589163Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:18.678039Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1010db2015e-68c6-4173-9723-ed8d4d526879 2025-06-24T10:25:18.678218Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3828s 2025-06-24T10:25:18.678843Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (95.41s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (re-run 2) (95.41s) === RUN TestTrustDomainValidation 2025-06-24T10:23:43.875571Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:25:19.305607Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m35.430024915s)' === 2025-06-24T10:25:19.305718Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:25:19.305765Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:25:19.305729Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:25:19.478284Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10117425831-8834-4ac4-a565-80081f03afec 2025-06-24T10:25:19.478642Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1726s 2025-06-24T10:25:19.479290Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:19.583783Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1010ec3d34c-ff74-45a3-9258-833d71432271 2025-06-24T10:25:19.583965Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2779s 2025-06-24T10:25:19.584703Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:19.665574Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1012137a21e-7726-45f5-ac94-5c9583eeb7a1 2025-06-24T10:25:19.665719Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3596s 2025-06-24T10:25:19.666454Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (96.45s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (re-run 2) (96.45s) 2025-06-24T10:25:20.323272Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-06-24T10:25:20.323360Z info tf === Suite "security_ca_custom_root" run time: 2m0.028650857s === 2025-06-24T10:25:20.323386Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-06-24T10:25:20.323959Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:20.323966Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:20.323972Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:20.693729Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:20Z/102604bb2dc-6788-49b8-9469-0a94e0c627e7 2025-06-24T10:25:20.693983Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3696s 2025-06-24T10:25:20.694722Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:20.842983Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:20Z/1024fae925f-00bf-40f7-b360-4dce67518876 2025-06-24T10:25:20.843102Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5187s 2025-06-24T10:25:20.843729Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:20.984654Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:20Z/1029ccd9604-88b5-4c32-b04a-67de3dc9397a 2025-06-24T10:25:20.984934Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6605s 2025-06-24T10:25:20.985621Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:21.685488Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:25:21.685586Z info tf === SUCCEEDED: Cleanup Istio in 80.826µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:tcp-mtls-pass-through\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3440157 2025-06-24T10:25:33.742018Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:25:33.742114Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:25:33.793300Z info tf === Test Framework Settings === 2025-06-24T10:25:33.793414Z info tf TestID: security_ca_custom_root RunID: 54d5ebb4-95ae-4505-863b-9bfd30d24342 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:25:33.793423Z info tf =============================== 2025-06-24T10:25:33.793696Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb 2025-06-24T10:25:33.793814Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:25:33.793836Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:25:33.794092Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:25:33.794109Z info tf === BEGIN: Building clusters === 2025-06-24T10:25:33.797747Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:25:33.797760Z info tf === DONE: Building clusters === 2025-06-24T10:25:33.797796Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:25:33.868751Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:25:33.868803Z info tf === SUCCESS: context setup function === 2025-06-24T10:25:33.868811Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:25:33.868831Z info tf === Istio Component Config === 2025-06-24T10:25:33.868898Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:25:33.868906Z info tf ================================ 2025-06-24T10:25:33.872512Z info tf skipping deployment as specified in the config 2025-06-24T10:25:33.872528Z info tf === SUCCEEDED: Deploy Istio in 3.714617ms [Suite=security_ca_custom_root]=== 2025-06-24T10:25:33.872537Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:25:33.920940Z info tf === SUCCEEDED: Create namespace echo1 in 48.393635ms === Generating RSA private key, 2048 bit long modulus (2 primes) **********************************************************************************************************************************************************************************************+++++ *************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb/_suite_context/ca-custom-root-2972442670/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb/_suite_context/ca-custom-root-2972442670/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************+++++ ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb/_suite_context/ca-custom-root-2972442670/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:25:36.363817Z info tf === BEGIN: Create namespace external === 2025-06-24T10:25:36.372854Z info tf === SUCCEEDED: Create namespace external in 9.055709ms === 2025-06-24T10:25:36.564584Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:25:36.624038Z error tf === FAILED: Deploy echo instances === 2025-06-24T10:25:36.624105Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb/_suite_context/Service.1861470676.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:25:36.624155Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:36.624199Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:36.624216Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:36.794464Z error klog an error occurred forwarding 43201 -> 15020: error forwarding port 15020 to pod c2d636b0c2ed4facf97e1852b4de4db3e8017176d05fb251eeb3239fd33e691d, uid : port forward into network namespace "/var/run/netns/c35314e2-f04d-41dc-871c-52ff6ff77b23": failed to connect to localhost:15020 inside namespace c2d636b0c2ed4facf97e1852b4de4db3e8017176d05fb251eeb3239fd33e691d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:25:36.796396Z error port forward failed: lost connection to pod 2025-06-24T10:25:36.796440Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43201/debug/ndsz": EOF 2025-06-24T10:25:36.797444Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43201/stats/prometheus": dial tcp 127.0.0.1:43201: connect: connection refused 2025-06-24T10:25:36.833645Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "client-v1-546c55dc9c-8ddkj" not found in the "echo1" namespace) 2025-06-24T10:25:36.966568Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-546c55dc9c-8ddkj/app for: pods "client-v1-546c55dc9c-8ddkj" not found 2025-06-24T10:25:36.984795Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-546c55dc9c-8ddkj/istio-proxy for: pods "client-v1-546c55dc9c-8ddkj" not found 2025-06-24T10:25:36.990694Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-546c55dc9c-8ddkj/istio-validation for: pods "client-v1-546c55dc9c-8ddkj" not found 2025-06-24T10:25:37.000626Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-64d65d6c8b-t549v/app for: pods "headless-v1-64d65d6c8b-t549v" not found 2025-06-24T10:25:37.010191Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-64d65d6c8b-t549v/istio-proxy for: pods "headless-v1-64d65d6c8b-t549v" not found 2025-06-24T10:25:37.027114Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-64d65d6c8b-t549v/istio-validation for: pods "headless-v1-64d65d6c8b-t549v" not found 2025-06-24T10:25:37.050885Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:34Z/1077e7aa74b-c686-4618-9f5f-6e26d099cb46 2025-06-24T10:25:37.051098Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4265s 2025-06-24T10:25:37.051883Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:37.208025Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:34Z/107280c88f8-8cd9-4a2f-b9c4-593d8ad627e0 2025-06-24T10:25:37.208167Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5836s 2025-06-24T10:25:37.208894Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:37.212397Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-v1-86cfb5b889-csx2d/istio-proxy for: pods "server-v1-86cfb5b889-csx2d" not found 2025-06-24T10:25:37.223625Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-v1-86cfb5b889-csx2d/istio-validation for: pods "server-v1-86cfb5b889-csx2d" not found 2025-06-24T10:25:37.244791Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-pt4z4/app for: pods "sotw-v1-65f898ccd5-pt4z4" not found 2025-06-24T10:25:37.256859Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-pt4z4/istio-proxy for: pods "sotw-v1-65f898ccd5-pt4z4" not found 2025-06-24T10:25:37.266124Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-65f898ccd5-pt4z4/istio-validation for: pods "sotw-v1-65f898ccd5-pt4z4" not found 2025-06-24T10:25:37.448625Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:34Z/107612f7028-e60d-459f-b02a-a8ce95109dbf 2025-06-24T10:25:37.448790Z info tf istioctl ([x internal-debug --all configz]): completed after 0.8242s 2025-06-24T10:25:37.449572Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:38.685477Z warn tf cluster/pod cluster-0/echo1/statefulset-v1-0 found warming resources (warming_state (listeners)) on attempt 1 2025-06-24T10:25:40.360208Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-06-24T10:25:40.366875Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb/_suite_context/Service.1861470676.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:25:40.366932Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb/_suite_context/Service.1861470676.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-06-24T10:25:40.366941Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-54d5ebb/_suite_context/Service.1861470676.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-06-24T10:25:40.367000Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:40.367001Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:40.367051Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:25:40.501873Z error klog an error occurred forwarding 46825 -> 15020: error forwarding port 15020 to pod c2d636b0c2ed4facf97e1852b4de4db3e8017176d05fb251eeb3239fd33e691d, uid : port forward into network namespace "/var/run/netns/c35314e2-f04d-41dc-871c-52ff6ff77b23": failed to connect to localhost:15020 inside namespace c2d636b0c2ed4facf97e1852b4de4db3e8017176d05fb251eeb3239fd33e691d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-06-24T10:25:40.503768Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:46825/debug/ndsz": EOF 2025-06-24T10:25:40.503991Z error port forward failed: lost connection to pod 2025-06-24T10:25:40.504428Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:46825/stats/prometheus": dial tcp 127.0.0.1:46825: connect: connection refused 2025-06-24T10:25:40.640034Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:34Z/107c71f259d-b1ab-4332-9d78-8ad0d0de88e0 2025-06-24T10:25:40.640233Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2729s 2025-06-24T10:25:40.641105Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:40.742101Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:34Z/1073cdffa8e-8f2d-4f4e-9945-426680454166 2025-06-24T10:25:40.742199Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3748s 2025-06-24T10:25:40.742935Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:40.864530Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:25:34Z/1078b3c803c-b9a4-4535-b083-eb6e4c6f0894 2025-06-24T10:25:40.864715Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4973s 2025-06-24T10:25:40.865303Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:41.177679Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:25:41.177759Z info tf === SUCCEEDED: Cleanup Istio in 74.954µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3441510 2025-06-24T10:25:52.459942Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:25:52.460004Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3/istio/out/linux_ppc64le does not exist 2025-06-24T10:25:52.493121Z info tf === Test Framework Settings === 2025-06-24T10:25:52.493220Z info tf TestID: security_ca_custom_root RunID: 19bbea01-dc73-435d-b5f0-713d1ba03792 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts GatewayConformanceStandardOnly: false 2025-06-24T10:25:52.493226Z info tf =============================== 2025-06-24T10:25:52.493400Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-19bbea0 2025-06-24T10:25:52.493491Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-06-24T10:25:52.493506Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-06-24T10:25:52.493525Z info tf Using KubeConfigs: [/root/upi/cB/auth/kubeconfig]. 2025-06-24T10:25:52.493533Z info tf === BEGIN: Building clusters === 2025-06-24T10:25:52.496748Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cB/auth/kubeconfig 2025-06-24T10:25:52.496756Z info tf === DONE: Building clusters === 2025-06-24T10:25:52.496786Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-06-24T10:25:52.561267Z info configmap istio-ca-root-cert is deleted 2025-06-24T10:25:52.561399Z info tf === SUCCESS: context setup function === 2025-06-24T10:25:52.561408Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-06-24T10:25:52.561425Z info tf === Istio Component Config === 2025-06-24T10:25:52.561487Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: 2025-06-24T10:25:52.561493Z info tf ================================ 2025-06-24T10:25:52.565086Z info tf skipping deployment as specified in the config 2025-06-24T10:25:52.565108Z info tf === SUCCEEDED: Deploy Istio in 3.695671ms [Suite=security_ca_custom_root]=== 2025-06-24T10:25:52.565117Z info tf === BEGIN: Create namespace echo1 === 2025-06-24T10:25:52.633110Z info tf === SUCCEEDED: Create namespace echo1 in 67.984894ms === Generating RSA private key, 2048 bit long modulus (2 primes) ***********************************************************************************+++++ ****************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-19bbea0/_suite_context/ca-custom-root-3022845548/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *************************************+++++ ************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-19bbea0/_suite_context/ca-custom-root-3022845548/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ************************************************************************+++++ ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-19bbea0/_suite_context/ca-custom-root-3022845548/workload-server-naked-foo-alt-cert.pem: OK 2025-06-24T10:25:53.900341Z info tf === BEGIN: Create namespace external === 2025-06-24T10:25:53.912702Z info tf === SUCCEEDED: Create namespace external in 12.415861ms === 2025-06-24T10:25:54.071328Z info tf === BEGIN: Deploy echo instances === 2025-06-24T10:25:55.451012Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.471590Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.477253Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.477918Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.508994Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.516849Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.541429Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.541750Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.541916Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.551298Z info klog Caches are synced for echo 2025-06-24T10:25:55.563697Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.572073Z info klog Caches are synced for echo 2025-06-24T10:25:55.572272Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.578307Z info klog Caches are synced for echo 2025-06-24T10:25:55.578369Z info klog Caches are synced for echo 2025-06-24T10:25:55.581704Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.582374Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.584161Z info klog Waiting for caches to sync for echo 2025-06-24T10:25:55.609347Z info klog Caches are synced for echo 2025-06-24T10:25:55.617507Z info klog Caches are synced for echo 2025-06-24T10:25:55.642252Z info klog Caches are synced for echo 2025-06-24T10:25:55.642339Z info klog Caches are synced for echo 2025-06-24T10:25:55.642355Z info klog Caches are synced for echo 2025-06-24T10:25:55.664336Z info klog Caches are synced for echo 2025-06-24T10:25:55.673144Z info klog Caches are synced for echo 2025-06-24T10:25:55.681939Z info klog Caches are synced for echo 2025-06-24T10:25:55.683401Z info klog Caches are synced for echo 2025-06-24T10:25:55.684702Z info klog Caches are synced for echo 2025-06-24T10:26:16.392582Z info tf === SUCCEEDED: Deploy echo instances in 22.321244453s === 2025-06-24T10:26:16.425869Z info tf === DONE: Setup: 'security_ca_custom_root' (23.929061049s) === 2025-06-24T10:26:16.425963Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:http-plaintext[allow] (re-run 2) (0.08s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:tcp-plaintext[allow] (re-run 2) (0.11s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:http-plaintext[allow] (re-run 2) (0.05s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:tcp-plaintext[allow] (re-run 2) (0.01s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:http-mtls[allow] (re-run 2) (0.06s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:tcp-mtls[allow] (re-run 2) (0.10s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:http-mtls[deny] (re-run 2) (0.37s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls[deny] (re-run 2) (0.05s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls-pass-through[deny] (re-run 2) (0.05s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-06-24T10:26:17.339309Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:26:47.340871Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.00155542s)' === 2025-06-24T10:26:47.340985Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:26:47.340969Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:26:47.341064Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:26:47.730416Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11814f90281-2eae-49d1-b0ec-15695556187a 2025-06-24T10:26:47.730728Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3894s 2025-06-24T10:26:47.731473Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:47.834463Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118498fa37f-0c80-437e-88da-314149243a3d 2025-06-24T10:26:47.834670Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4933s 2025-06-24T10:26:47.835536Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:47.990584Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11869ac6208-cf93-4ed2-a89e-c7668f19ce72 2025-06-24T10:26:47.990906Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6496s 2025-06-24T10:26:47.991275Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (31.25s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 2) (31.25s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] 2025-06-24T10:26:48.591243Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-06-24T10:27:06.384590Z error port forward failed: lost connection to pod 2025-06-24T10:27:06.678581Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-t8tmt port forward terminated 2025-06-24T10:27:06.716313Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-t8tmt port forward reconnect success 2025-06-24T10:27:06.974761Z error port forward failed: lost connection to pod 2025-06-24T10:27:07.043271Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-qvxzg port forward terminated 2025-06-24T10:27:07.046373Z error port forward failed: lost connection to pod 2025-06-24T10:27:07.081153Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-qvxzg port forward reconnect success 2025-06-24T10:27:07.165460Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-z9694 port forward terminated 2025-06-24T10:27:07.210050Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-z9694 port forward reconnect success 2025-06-24T10:27:08.068936Z error port forward failed: lost connection to pod 2025-06-24T10:27:08.282774Z warn pod: external/external-v1-f9d74566f-x8q94 port forward terminated 2025-06-24T10:27:08.316018Z warn pod: external/external-v1-f9d74566f-x8q94 port forward reconnect success 2025-06-24T10:27:08.838172Z error port forward failed: lost connection to pod 2025-06-24T10:27:09.229256Z warn pod: echo1/proxyless-grpc-v1-8996df688-hpxhh port forward terminated 2025-06-24T10:27:09.261507Z warn pod: echo1/proxyless-grpc-v1-8996df688-hpxhh port forward reconnect success 2025-06-24T10:27:18.480015Z error port forward failed: lost connection to pod test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:27:18.592762Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.001513868s)' === 2025-06-24T10:27:18.592882Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:27:18.592886Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:27:18.592931Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:27:18.827593Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118ed67cda2-26b3-42f4-bc07-5427550d4cde 2025-06-24T10:27:18.827960Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2347s 2025-06-24T10:27:18.828571Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:18.917343Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1189bf3dee4-047d-4b76-9420-b9ebfce78e19 2025-06-24T10:27:18.917544Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3242s 2025-06-24T10:27:18.918194Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:18.950523Z warn pod: echo1/sotw-v1-65f898ccd5-2lp6d port forward terminated 2025-06-24T10:27:19.007046Z warn pod: echo1/sotw-v1-65f898ccd5-2lp6d port forward reconnect success 2025-06-24T10:27:19.013490Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1180c219b1c-2563-4067-94c2-75c9cdd6d53a 2025-06-24T10:27:19.013680Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4203s 2025-06-24T10:27:19.014161Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (30.90s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (re-run 2) (30.90s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] 2025-06-24T10:27:19.488257Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-06-24T10:27:19.666199Z error port forward failed: lost connection to pod 2025-06-24T10:27:19.692927Z error port forward failed: lost connection to pod 2025-06-24T10:27:19.727786Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:27:19.760839Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:27:19.906498Z error port forward failed: lost connection to pod 2025-06-24T10:27:20.049992Z warn pod: echo1/a-v1-78f49b749b-mp5k2 port forward terminated 2025-06-24T10:27:20.081360Z warn pod: echo1/a-v1-78f49b749b-mp5k2 port forward reconnect success 2025-06-24T10:27:20.165676Z warn pod: echo1/headless-v1-64d65d6c8b-76jtq port forward terminated 2025-06-24T10:27:20.198658Z warn pod: echo1/headless-v1-64d65d6c8b-76jtq port forward reconnect success 2025-06-24T10:27:20.543999Z error port forward failed: lost connection to pod 2025-06-24T10:27:20.722624Z error port forward failed: lost connection to pod 2025-06-24T10:27:20.812698Z warn pod: echo1/server-v1-86cfb5b889-r86pq port forward terminated 2025-06-24T10:27:20.841229Z warn pod: echo1/server-v1-86cfb5b889-r86pq port forward reconnect success 2025-06-24T10:27:21.200464Z warn pod: echo1/b-v1-7bcb76fc6f-ghzdk port forward terminated 2025-06-24T10:27:21.232697Z warn pod: echo1/b-v1-7bcb76fc6f-ghzdk port forward reconnect success 2025-06-24T10:27:21.430231Z error port forward failed: lost connection to pod 2025-06-24T10:27:21.675373Z warn pod: echo1/c-v1-6975698f5d-s8mwt port forward terminated 2025-06-24T10:27:21.712123Z warn pod: echo1/c-v1-6975698f5d-s8mwt port forward reconnect success 2025-06-24T10:27:22.882120Z error port forward failed: lost connection to pod 2025-06-24T10:27:23.009001Z warn pod: echo1/client-v1-546c55dc9c-spwlj port forward terminated 2025-06-24T10:27:23.041485Z warn pod: echo1/client-v1-546c55dc9c-spwlj port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.79:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.79:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:27:49.488947Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000686659s)' === 2025-06-24T10:27:49.489038Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:27:49.489043Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:27:49.489091Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:27:49.643280Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11846e9a4b6-b197-4323-866f-3b881911b253 2025-06-24T10:27:49.643532Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1541s 2025-06-24T10:27:49.644114Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:49.730059Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118f7dc2eda-7e74-40bb-af2c-af5be7ee499b 2025-06-24T10:27:49.730385Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2409s 2025-06-24T10:27:49.731099Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:49.802340Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118210a2c66-7ad0-4b62-a2d2-4e532a240054 2025-06-24T10:27:49.802487Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3130s 2025-06-24T10:27:49.803082Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (30.80s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 2) (30.80s) === RUN TestTrustDomainValidation/From_primary-0 2025-06-24T10:26:16.465146Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:27:50.289877Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m33.824722936s)' === 2025-06-24T10:27:50.290021Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:27:50.290041Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:27:50.290064Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:27:50.455757Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1187ba782c6-fee8-40c3-a520-4d5b5b108f6f 2025-06-24T10:27:50.456140Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1659s 2025-06-24T10:27:50.456725Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:50.538201Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118ecd0aa94-dfb8-4bfc-8171-fb207827761e 2025-06-24T10:27:50.538418Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2481s 2025-06-24T10:27:50.539006Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:50.635334Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1189831f22f-acc5-4ba7-b4ba-3757c87ce67c 2025-06-24T10:27:50.635555Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3452s 2025-06-24T10:27:50.636039Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (94.74s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (re-run 2) (94.74s) === RUN TestTrustDomainValidation 2025-06-24T10:26:16.426659Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:27:51.205724Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m34.779051747s)' === 2025-06-24T10:27:51.205886Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:27:51.205900Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:27:51.205913Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:27:51.418792Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118434a4f51-aeab-4e2d-a67e-859e87cddea0 2025-06-24T10:27:51.419035Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2128s 2025-06-24T10:27:51.419824Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:51.529897Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118492ab166-e4e1-4162-ac82-a38383798230 2025-06-24T10:27:51.530215Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3240s 2025-06-24T10:27:51.530813Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:51.618302Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11874b632f4-1163-4bb2-9215-f5cb01dc56d8 2025-06-24T10:27:51.618518Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4123s 2025-06-24T10:27:51.619084Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (95.77s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (re-run 2) (95.77s) 2025-06-24T10:27:52.200316Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-06-24T10:27:52.200371Z info tf === Suite "security_ca_custom_root" run time: 1m59.703597592s === 2025-06-24T10:27:52.200387Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-06-24T10:27:52.200720Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-06-24T10:27:52.200729Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-06-24T10:27:52.200795Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-06-24T10:27:52.525214Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:27:52Z/119da074f9b-d99a-4dcd-9723-7cabb2132757 2025-06-24T10:27:52.525640Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3246s 2025-06-24T10:27:52.526145Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:52.655888Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:27:52Z/1191ce1ae4e-a53a-455a-833b-688e83de8dcf 2025-06-24T10:27:52.655995Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4549s 2025-06-24T10:27:52.656631Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:52.738561Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:27:52Z/11920bf168d-49cb-432b-8b60-51a724abde83 2025-06-24T10:27:52.738774Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5377s 2025-06-24T10:27:52.739474Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:53.570921Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-06-24T10:27:53.571095Z info tf === SUCCEEDED: Cleanup Istio in 161.495µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root === Failed === FAIL: tests/integration/security/ca_custom_root TestSecureNaming (11.87s) 2025-06-24T10:13:52.017435Z info tf === BEGIN: Test: 'security_ca_custom_root[TestSecureNaming]' === secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:114: retry.UntilSuccessOrFail: timeout while waiting after 10 attempts (last error: CA certificate (-----BEGIN CERTIFICATE----- MIIC/TCCAeWgAwIBAgIRAOultIFQpmMoQ3Ba4vBy3uowDQYJKoZIhvcNAQELBQAw GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yNTA2MjMxNzUyMTFaFw0zNTA2 MjExNzUyMTFaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3 DQEBAQUAA4IBDwAwggEKAoIBAQDD61K6//kQ50joJsVYbHd4cHQMVhi0HNh4K71M VAuIRgtcV/1g5lsTSzlytjzYSgW0VTAP1VFBQqNVUo7pmQNLEZaWHTpi/8/I5N5D 5/hztT8WqCCaWoyK2nh+fQX+FDzPTI/O8nwIx/FRGXflx2rIpiDrB3kX3ERV5l9d tKsXMIbHVJQ7xn8c24As0kdBzZKmzbHmzaLgpNHftqPiuD0PYLhwCHvQvIXLapuW I3LvuPoRGtnxPTSh8B4wnRz9LEfjmDVJZZ89jDN1Qubj3n4OsSLUTLsJ5bFgeTmO 7hghz4zM8zM6ZAY5zqFmw2zcmAlFrACDKtIMf6ONd71BArkFAgMBAAGjQjBAMA4G A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT0Czg2lEDA pSVzoN5aJDxtUzmFwzANBgkqhkiG9w0BAQsFAAOCAQEAvSGwExBY/OLSdjv10G0e T58ioViIHqsCDR8JC23xJstoHg/RGQbGZsjiTIAv6uifa3uOWmfO2bhoZTeBtAMy R/LhgKQuukZltVRoLyp7bCALB0ZLNoPrbrgR83SEWtoCEQ9lW7iT+f/MKIMt9qja g0dURbkIAasOF58uTSgSOwEJw2M5kmq6//K/krVWM44FjsMdwgyDw5s6EGRU/jB8 6OK8czEjuPLjAMbYIQKTaorEt5zu9PFFJyLTAw8fw8SXoqDgS6/1zzWdv3XOwRR5 dIv2JC9JiGi6ch9ICpItdK9pUw4iouKfUw/opsUHjayU8FVqWtzVxZkjJwIVdZz1 1A== -----END CERTIFICATE----- ) not matching plugin cert (-----BEGIN CERTIFICATE----- MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN 8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a 3g== -----END CERTIFICATE----- )) 2025-06-24T10:14:02.026331Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestSecureNaming] (10.008893754s)' === 2025-06-24T10:14:02.026426Z error tf === Dumping Namespace external State for TestSecureNaming... 2025-06-24T10:14:02.026421Z error tf === Dumping Istio Deployment State for TestSecureNaming... 2025-06-24T10:14:02.026451Z error tf === Dumping Namespace echo1 State for TestSecureNaming... 2025-06-24T10:14:02.338526Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:13:52Z/16187d2201-0932-442a-a2db-eff7aab8b168 2025-06-24T10:14:02.338766Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3120s 2025-06-24T10:14:02.339390Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:02.454251Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:13:52Z/16b1be7fae-3ec2-44c3-aacd-7bad21bcca72 2025-06-24T10:14:02.454481Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4277s 2025-06-24T10:14:02.455275Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:02.531891Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:13:52Z/1600f3c1f6-a63e-4dcb-9787-6ab12681f313 2025-06-24T10:14:02.532117Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5053s 2025-06-24T10:14:02.532697Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:03.345047Z error tf === Dumping Namespace istio-system State for TestSecureNaming... === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (33.16s) 2025-06-24T10:14:03.901376Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === 2025-06-24T10:14:28.009338Z error port forward failed: lost connection to pod 2025-06-24T10:14:28.317020Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward terminated 2025-06-24T10:14:28.355160Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward reconnect success 2025-06-24T10:14:28.747815Z error port forward failed: lost connection to pod 2025-06-24T10:14:28.762393Z error port forward failed: lost connection to pod 2025-06-24T10:14:28.782842Z warn pod: echo1/naked-v1-757556fc5c-fx9tx port forward terminated 2025-06-24T10:14:28.818217Z warn pod: echo1/naked-v1-757556fc5c-fx9tx port forward reconnect success 2025-06-24T10:14:28.885331Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward terminated 2025-06-24T10:14:28.923556Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward reconnect success 2025-06-24T10:14:29.654778Z error port forward failed: lost connection to pod 2025-06-24T10:14:29.846349Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward terminated 2025-06-24T10:14:29.882781Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward reconnect success 2025-06-24T10:14:30.289606Z error port forward failed: lost connection to pod 2025-06-24T10:14:30.453789Z warn pod: external/external-v1-f9d74566f-lg7nm port forward terminated 2025-06-24T10:14:30.508342Z warn pod: external/external-v1-f9d74566f-lg7nm port forward reconnect success test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.3.45:48546->172.30.13.26:443: read: connection reset by peer 2025-06-24T10:14:33.902412Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.001025957s)' === 2025-06-24T10:14:33.902492Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:14:33.902550Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:14:33.902520Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:14:36.150952Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/178e6e3d25-c3e5-465e-8948-c624f92f6b20 2025-06-24T10:14:36.151163Z info tf istioctl ([x internal-debug --all mcsz]): completed after 2.2481s 2025-06-24T10:14:36.151753Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:36.251937Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17e1e447b2-03a0-4675-a65d-bfa699ca8e86 2025-06-24T10:14:36.252218Z info tf istioctl ([x internal-debug --all configz]): completed after 2.3491s 2025-06-24T10:14:36.252695Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:36.367824Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17f0ea4a98-ac41-4245-bbaa-b3ca4f977bf8 2025-06-24T10:14:36.368020Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.4649s 2025-06-24T10:14:36.368674Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0 (34.03s) 2025-06-24T10:14:03.901130Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-06-24T10:14:37.116500Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (33.215370633s)' === 2025-06-24T10:14:37.116538Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:14:37.116562Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:14:37.116535Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:14:37.296402Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17555f2755-71e0-44fe-a7b8-4c082581d274 2025-06-24T10:14:37.296541Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1796s 2025-06-24T10:14:37.297000Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:37.408221Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17e73a9598-ad8f-473d-82c2-3591b833d397 2025-06-24T10:14:37.408422Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2915s 2025-06-24T10:14:37.409072Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:37.480181Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17d58e1f36-c692-4992-b075-9ec5b69bb963 2025-06-24T10:14:37.480323Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3634s 2025-06-24T10:14:37.480983Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming (34.89s) 2025-06-24T10:14:03.882807Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-06-24T10:14:37.929664Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (34.046845913s)' === 2025-06-24T10:14:37.929761Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:14:37.929758Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:14:37.929782Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:14:38.122609Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/178f00552d-fc78-4c1f-94d2-47143a704abb 2025-06-24T10:14:38.122796Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1928s 2025-06-24T10:14:38.123525Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:38.250687Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17dabebecb-83a3-48d1-80e9-04dd1a0f2388 2025-06-24T10:14:38.251014Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3210s 2025-06-24T10:14:38.251651Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:14:38.319501Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:04Z/17092a6082-d8c0-41cd-9e3f-eeb98b91b9a4 2025-06-24T10:14:38.319585Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3896s 2025-06-24T10:14:38.320120Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (30.99s) 2025-06-24T10:14:44.408356Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === 2025-06-24T10:14:49.092340Z error port forward failed: lost connection to pod 2025-06-24T10:14:49.531962Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward terminated 2025-06-24T10:14:49.568985Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward reconnect success 2025-06-24T10:14:49.734066Z error port forward failed: lost connection to pod 2025-06-24T10:14:49.784788Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward terminated 2025-06-24T10:14:49.820394Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward reconnect success 2025-06-24T10:14:50.103340Z error port forward failed: lost connection to pod 2025-06-24T10:14:50.146016Z error port forward failed: lost connection to pod 2025-06-24T10:14:50.595912Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward terminated 2025-06-24T10:14:50.600152Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward terminated 2025-06-24T10:14:50.636416Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward reconnect success 2025-06-24T10:14:50.637159Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward reconnect success 2025-06-24T10:14:50.711747Z error port forward failed: lost connection to pod 2025-06-24T10:14:51.157577Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward terminated 2025-06-24T10:14:51.189845Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward reconnect success 2025-06-24T10:14:51.561966Z error port forward failed: lost connection to pod 2025-06-24T10:14:52.006049Z error port forward failed: lost connection to pod 2025-06-24T10:14:52.045847Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:14:52.081007Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:14:52.322224Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward terminated 2025-06-24T10:14:52.359471Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:15:14.409532Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.001171928s)' === 2025-06-24T10:15:14.409630Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:15:14.409651Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:15:14.409686Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:15:14.630081Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18dc276bcc-8596-43ac-98de-fd7c4ee5bd5b 2025-06-24T10:15:14.630376Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2203s 2025-06-24T10:15:14.631060Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:14.726373Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/1840095c63-a037-4787-af51-3ab478e7210a 2025-06-24T10:15:14.726582Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3165s 2025-06-24T10:15:14.727349Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:14.797759Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/183fe7cf00-68e3-49d4-a74a-0c718d02e9a8 2025-06-24T10:15:14.797928Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3878s 2025-06-24T10:15:14.798478Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (30.90s) 2025-06-24T10:15:15.395261Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-06-24T10:15:34.364386Z error port forward failed: lost connection to pod 2025-06-24T10:15:34.817118Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward terminated 2025-06-24T10:15:34.854764Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-2xfrl port forward reconnect success 2025-06-24T10:15:34.937153Z error port forward failed: lost connection to pod 2025-06-24T10:15:35.384818Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward terminated 2025-06-24T10:15:35.430086Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-j7j56 port forward reconnect success 2025-06-24T10:15:35.889292Z error port forward failed: lost connection to pod 2025-06-24T10:15:36.346523Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward terminated 2025-06-24T10:15:36.395306Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-4gdpk port forward reconnect success 2025-06-24T10:15:36.512549Z error port forward failed: lost connection to pod 2025-06-24T10:15:36.953452Z warn pod: external/external-v1-f9d74566f-lg7nm port forward terminated 2025-06-24T10:15:36.994770Z warn pod: external/external-v1-f9d74566f-lg7nm port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:15:45.396092Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.000825202s)' === 2025-06-24T10:15:45.396183Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:15:45.396202Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:15:45.396186Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:15:45.597985Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18b3574d15-6ae3-4ea9-8764-6675684100fc 2025-06-24T10:15:45.598244Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2017s 2025-06-24T10:15:45.598823Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:45.693501Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18bb468c7d-ed12-472f-85c8-8d0d926f1fb6 2025-06-24T10:15:45.693714Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2972s 2025-06-24T10:15:45.694323Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:15:45.782478Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/183fb5381c-fa39-4aab-8a80-e7e606b2689b 2025-06-24T10:15:45.782650Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3860s 2025-06-24T10:15:45.783183Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.13s) 2025-06-24T10:15:46.295761Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-06-24T10:15:48.834040Z error port forward failed: lost connection to pod 2025-06-24T10:15:49.287530Z warn pod: echo1/proxyless-grpc-v1-8996df688-bdq7m port forward terminated 2025-06-24T10:15:49.324333Z warn pod: echo1/proxyless-grpc-v1-8996df688-bdq7m port forward reconnect success 2025-06-24T10:15:50.229775Z error port forward failed: lost connection to pod 2025-06-24T10:15:50.645613Z warn pod: echo1/client-v1-546c55dc9c-9lmdp port forward terminated 2025-06-24T10:15:50.683390Z warn pod: echo1/client-v1-546c55dc9c-9lmdp port forward reconnect success 2025-06-24T10:15:55.573122Z error port forward failed: lost connection to pod 2025-06-24T10:15:55.832555Z error port forward failed: lost connection to pod 2025-06-24T10:15:56.032471Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward terminated 2025-06-24T10:15:56.067107Z warn pod: echo1/c-v1-6975698f5d-q5jkr port forward reconnect success 2025-06-24T10:15:56.284390Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward terminated 2025-06-24T10:15:56.322522Z warn pod: echo1/server-v1-86cfb5b889-rlhjn port forward reconnect success 2025-06-24T10:15:56.641544Z error port forward failed: lost connection to pod 2025-06-24T10:15:56.641591Z error port forward failed: lost connection to pod 2025-06-24T10:15:57.095443Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward terminated 2025-06-24T10:15:57.099588Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward terminated 2025-06-24T10:15:57.135645Z warn pod: echo1/headless-v1-64d65d6c8b-wdr9s port forward reconnect success 2025-06-24T10:15:57.136318Z warn pod: echo1/a-v1-78f49b749b-crwrb port forward reconnect success 2025-06-24T10:15:57.195777Z error port forward failed: lost connection to pod 2025-06-24T10:15:57.657207Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward terminated 2025-06-24T10:15:57.693946Z warn pod: echo1/b-v1-7bcb76fc6f-d8h2w port forward reconnect success 2025-06-24T10:15:58.087350Z error port forward failed: lost connection to pod 2025-06-24T10:15:58.363625Z error port forward failed: lost connection to pod 2025-06-24T10:15:58.545269Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:15:58.576470Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:15:58.822189Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward terminated 2025-06-24T10:15:58.860960Z warn pod: echo1/sotw-v1-65f898ccd5-mh5qc port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.42:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.42:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:16:16.296571Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000808771s)' === 2025-06-24T10:16:16.296676Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:16:16.296676Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:16:16.296711Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:16:16.700107Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/1893e07ce5-132c-4356-b544-bc911123bafe 2025-06-24T10:16:16.700363Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4032s 2025-06-24T10:16:16.700954Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:16.776945Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/186d86ec19-150f-45ab-891b-38b0da711b61 2025-06-24T10:16:16.777138Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4800s 2025-06-24T10:16:16.777686Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:16.862039Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/189638363d-861f-4e0f-8eee-f7f52f697505 2025-06-24T10:16:16.862157Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5650s 2025-06-24T10:16:16.862762Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (99.51s) 2025-06-24T10:14:38.791249Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:16:17.424781Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m38.633528973s)' === 2025-06-24T10:16:17.424862Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:16:17.424873Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:16:17.424915Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:16:17.595246Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/186c2ee1ba-ead3-43b9-900f-7d77589db1ef 2025-06-24T10:16:17.595516Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1705s 2025-06-24T10:16:17.596322Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:17.699698Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/182c4f0efa-f256-4052-aa4d-aca566d488a8 2025-06-24T10:16:17.699786Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2748s 2025-06-24T10:16:17.700408Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:17.771820Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/184d5964a5-df79-4c2c-98b1-a1e7bb889cac 2025-06-24T10:16:17.771955Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3469s 2025-06-24T10:16:17.772692Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (100.52s) 2025-06-24T10:14:38.771055Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:16:18.301377Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m39.530317091s)' === 2025-06-24T10:16:18.301484Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:16:18.301502Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:16:18.301484Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:16:18.509441Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/187d3a85de-8486-48a3-9064-8c0ca4dfa018 2025-06-24T10:16:18.509708Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2080s 2025-06-24T10:16:18.510435Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:18.587206Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/1884c9bebd-8890-4c37-a67e-ba3c06275fad 2025-06-24T10:16:18.587397Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2857s 2025-06-24T10:16:18.588125Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:16:18.699634Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:14:38Z/18f5961ac1-b03a-4e86-b1b4-b1e141643551 2025-06-24T10:16:18.699772Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3980s 2025-06-24T10:16:18.700547Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 1) (31.17s) 2025-06-24T10:17:23.918025Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.21.3.128:47180->172.30.198.181:443: read: connection reset by peer 2025-06-24T10:17:53.924448Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006418727s)' === 2025-06-24T10:17:53.924513Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:17:53.924520Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:17:53.924550Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:17:54.252196Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/354a17000a-f8c1-40e5-a03c-12406f2cf890 2025-06-24T10:17:54.252470Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3277s 2025-06-24T10:17:54.254035Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:54.379799Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35cf8c0d85-1883-424c-8aba-66260fa8dc7b 2025-06-24T10:17:54.379941Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4552s 2025-06-24T10:17:54.380781Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:54.472243Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/359c2940d9-afa3-4492-9b36-3f69f7dba078 2025-06-24T10:17:54.472441Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5476s 2025-06-24T10:17:54.472940Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 1) (32.41s) 2025-06-24T10:17:23.917793Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-06-24T10:17:55.085042Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (31.167241292s)' === 2025-06-24T10:17:55.085152Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:17:55.085154Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:17:55.085171Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:17:55.336675Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35238225c5-4d29-4686-abcb-3f1790e68b3e 2025-06-24T10:17:55.336961Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2516s 2025-06-24T10:17:55.337428Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:55.446369Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/359bc4c8ea-8c2e-4c1f-864d-8c92e54a3415 2025-06-24T10:17:55.446548Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3612s 2025-06-24T10:17:55.447262Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:55.530884Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35620451f7-c105-4875-9f0d-bd29d22731ef 2025-06-24T10:17:55.531103Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4458s 2025-06-24T10:17:55.531730Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming (re-run 1) (33.80s) 2025-06-24T10:17:23.900535Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-06-24T10:17:56.324206Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.423663615s)' === 2025-06-24T10:17:56.324335Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:17:56.324349Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:17:56.324367Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:17:56.522168Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/355700bc02-4828-4ebb-b45e-d398927b9189 2025-06-24T10:17:56.522483Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1979s 2025-06-24T10:17:56.522801Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:56.615289Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/350c87d4e4-019d-4cf5-a9b0-4e73dddeaa92 2025-06-24T10:17:56.615467Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2909s 2025-06-24T10:17:56.616035Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:17:56.744268Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:17:24Z/35182b1fe9-034f-45ae-8088-f557640f7912 2025-06-24T10:17:56.744452Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4199s 2025-06-24T10:17:56.745137Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 1) (31.18s) 2025-06-24T10:18:55.535260Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:19:25.536019Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000754234s)' === 2025-06-24T10:19:25.536102Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:19:25.536111Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:19:25.536155Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:19:25.878004Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/491bdc585c-2f0b-4419-991f-c194b50c0859 2025-06-24T10:19:25.878398Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3421s 2025-06-24T10:19:25.878901Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:26.048371Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/495f55e95e-189b-4e83-94a1-0a54b4c719e6 2025-06-24T10:19:26.048581Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5122s 2025-06-24T10:19:26.049288Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:26.174397Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4975262ce6-cfe1-42a4-a884-64f07b233b5b 2025-06-24T10:19:26.174568Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6382s 2025-06-24T10:19:26.175140Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (re-run 1) (32.33s) 2025-06-24T10:18:55.534280Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:19:26.712998Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (31.178715768s)' === 2025-06-24T10:19:26.713078Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:19:26.713096Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:19:26.713093Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:19:27.216184Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/49244bac22-42d0-4b28-978f-0157e70d831a 2025-06-24T10:19:27.216642Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5033s 2025-06-24T10:19:27.217124Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:27.321518Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4915a1b3fb-0225-4481-9697-e5b61c0a5e75 2025-06-24T10:19:27.321666Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6084s 2025-06-24T10:19:27.322266Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:27.417191Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4917560590-429c-46a0-bd3d-67ba090bc097 2025-06-24T10:19:27.417310Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7040s 2025-06-24T10:19:27.417896Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (re-run 1) (33.54s) 2025-06-24T10:18:55.515499Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:19:27.859696Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (32.344188797s)' === 2025-06-24T10:19:27.859831Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:19:27.859846Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:19:27.859824Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:19:28.044510Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/4945fd248e-0d79-4058-bee9-0cb5a3a42ddf 2025-06-24T10:19:28.044716Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1846s 2025-06-24T10:19:28.045137Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:28.189161Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/490a4ee036-5efb-41c5-b7d7-de03f0ee1b88 2025-06-24T10:19:28.189264Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3291s 2025-06-24T10:19:28.190096Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:19:28.306787Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:18:55Z/49c217e7bc-1a08-44e9-abab-4672d4b84fba 2025-06-24T10:19:28.306924Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4468s 2025-06-24T10:19:28.307589Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 1) (31.18s) 2025-06-24T10:20:30.738992Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.21.3.142:9000': call failed from naked (cluster=cluster-0) to tcp://172.21.3.142:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:21:00.739881Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.00088158s)' === 2025-06-24T10:21:00.740022Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:21:00.739988Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:21:00.740012Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:21:01.063645Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68aa30caef-f0b3-41f1-91a2-e9c61c11a8df 2025-06-24T10:21:01.063917Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3231s 2025-06-24T10:21:01.064547Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:01.155390Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68c9414860-ead5-4dcf-9033-490578e0519e 2025-06-24T10:21:01.155670Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4148s 2025-06-24T10:21:01.156307Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:01.235100Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68a7d86e13-eca7-4b06-bcef-db8a671bb7b1 2025-06-24T10:21:01.235249Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4944s 2025-06-24T10:21:01.235986Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (re-run 1) (32.24s) 2025-06-24T10:20:30.738126Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:21:01.921068Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (31.182935092s)' === 2025-06-24T10:21:01.921196Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:21:01.921198Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:21:01.921222Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:21:02.111299Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/680ee68527-6a9f-4c50-9b8d-40feef3c56c5 2025-06-24T10:21:02.111613Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1902s 2025-06-24T10:21:02.112013Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:02.199728Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/6897e73ddf-d3a4-4a02-9183-03b10b716d0e 2025-06-24T10:21:02.199910Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2785s 2025-06-24T10:21:02.200607Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:02.349467Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68bfac5bbb-fa1a-49e2-bff9-084bdbc138e4 2025-06-24T10:21:02.349680Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4282s 2025-06-24T10:21:02.351904Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (re-run 1) (33.42s) 2025-06-24T10:20:30.689905Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:21:02.981236Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (32.291319037s)' === 2025-06-24T10:21:02.981379Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:21:02.981397Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:21:02.981387Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:21:03.219915Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/6880b502a3-ce16-45b2-9e75-8ec0ac0695b3 2025-06-24T10:21:03.220222Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2386s 2025-06-24T10:21:03.220598Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:03.345260Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/68506004ce-ad2e-4d4f-aea4-b554a34bc2fd 2025-06-24T10:21:03.345401Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3637s 2025-06-24T10:21:03.345990Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:21:03.442976Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:20:30Z/683e933305-9c3f-4b98-9113-c771256984d5 2025-06-24T10:21:03.443114Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4614s 2025-06-24T10:21:03.443685Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 2) (31.99s) 2025-06-24T10:22:12.346184Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.3.67:41246->172.30.112.136:443: read: connection reset by peer 2025-06-24T10:22:42.353050Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006863046s)' === 2025-06-24T10:22:42.353100Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:22:42.353094Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:22:42.353116Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-06-24T10:22:43.448918Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86af6823cc-f3bf-4732-b748-17730691dbe0 2025-06-24T10:22:43.449168Z info tf istioctl ([x internal-debug --all configz]): completed after 1.0957s 2025-06-24T10:22:43.449724Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:43.539562Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86bad295bb-b242-4abc-bc3a-4577b1f4a535 2025-06-24T10:22:43.539726Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1862s 2025-06-24T10:22:43.540254Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:43.630966Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/865e41ee7a-f616-44f0-8db6-9f105715b1c2 2025-06-24T10:22:43.631114Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.2776s 2025-06-24T10:22:43.631650Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 2) (33.54s) 2025-06-24T10:22:12.345991Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-06-24T10:22:44.364746Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (32.018754562s)' === 2025-06-24T10:22:44.364796Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:22:44.364805Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:22:44.364835Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-06-24T10:22:45.001870Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/860b15012e-bf60-414b-b2fa-111d8bab308f 2025-06-24T10:22:45.002120Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6370s 2025-06-24T10:22:45.002734Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:45.203131Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86892a7d20-2d11-435e-a231-d0aaec034636 2025-06-24T10:22:45.203262Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8381s 2025-06-24T10:22:45.203864Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:45.293394Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/869b713234-87ba-43e9-bed1-e6705cd19910 2025-06-24T10:22:45.293563Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9283s 2025-06-24T10:22:45.294090Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming (re-run 2) (34.62s) 2025-06-24T10:22:12.329300Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-06-24T10:22:45.881530Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (33.552224275s)' === 2025-06-24T10:22:45.881599Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:22:45.881612Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:22:45.881617Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-06-24T10:22:46.111697Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86c40cbc5e-7c67-4ec5-be81-c60644350851 2025-06-24T10:22:46.111953Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2302s 2025-06-24T10:22:46.112421Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:46.225778Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/86a72730d3-3f97-414e-92e7-c86666ea27aa 2025-06-24T10:22:46.225883Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3441s 2025-06-24T10:22:46.226365Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:22:46.325864Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:22:12Z/8605827fe3-00a6-436b-b4d8-ef7699d6e450 2025-06-24T10:22:46.325938Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4441s 2025-06-24T10:22:46.326556Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 2) (31.38s) 2025-06-24T10:23:44.944909Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:24:14.946250Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.001336344s)' === 2025-06-24T10:24:14.946332Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:24:14.946330Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:24:14.946362Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:24:15.226949Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1011beb4930-5e73-43c8-90c5-0bc85f9aa040 2025-06-24T10:24:15.227328Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2807s 2025-06-24T10:24:15.227864Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:15.405526Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1019f6f97a0-dba1-436d-86d5-2229c058c0ac 2025-06-24T10:24:15.405822Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4591s 2025-06-24T10:24:15.406912Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:15.516013Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10123f09d91-3788-4948-bca7-f97c217b050d 2025-06-24T10:24:15.516231Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5695s 2025-06-24T10:24:15.516677Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (re-run 2) (31.03s) 2025-06-24T10:24:16.326302Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-06-24T10:24:35.180024Z error port forward failed: lost connection to pod 2025-06-24T10:24:35.603733Z warn pod: external/external-v1-f9d74566f-59fvt port forward terminated 2025-06-24T10:24:35.684012Z warn pod: external/external-v1-f9d74566f-59fvt port forward reconnect success 2025-06-24T10:24:36.003721Z error port forward failed: lost connection to pod 2025-06-24T10:24:36.204263Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-pm52m port forward terminated 2025-06-24T10:24:36.236700Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-pm52m port forward reconnect success 2025-06-24T10:24:36.256334Z error port forward failed: lost connection to pod 2025-06-24T10:24:36.258577Z error port forward failed: lost connection to pod 2025-06-24T10:24:36.725472Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-jg9qk port forward terminated 2025-06-24T10:24:36.725423Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-zl8xx port forward terminated 2025-06-24T10:24:36.760903Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-zl8xx port forward reconnect success 2025-06-24T10:24:36.763549Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-jg9qk port forward reconnect success 2025-06-24T10:24:39.207463Z error port forward failed: lost connection to pod 2025-06-24T10:24:39.428318Z warn pod: echo1/proxyless-grpc-v1-8996df688-kk77f port forward terminated 2025-06-24T10:24:39.465065Z warn pod: echo1/proxyless-grpc-v1-8996df688-kk77f port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:24:46.327056Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.000751017s)' === 2025-06-24T10:24:46.327134Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:24:46.327128Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:24:46.327201Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:24:46.659123Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1019c01650a-51f5-47ee-880f-a51069feb1da 2025-06-24T10:24:46.659385Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3317s 2025-06-24T10:24:46.660035Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:46.813668Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101fbb18efc-d664-45ed-89c3-896dd8c6c77a 2025-06-24T10:24:46.813792Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4861s 2025-06-24T10:24:46.814523Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:24:46.949459Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101ad90b4d2-1a1a-48ef-9580-0d80c4bdd503 2025-06-24T10:24:46.949615Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6219s 2025-06-24T10:24:46.950497Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 2) (30.94s) 2025-06-24T10:24:47.356232Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-06-24T10:24:47.496517Z error port forward failed: lost connection to pod 2025-06-24T10:24:47.819520Z error port forward failed: lost connection to pod 2025-06-24T10:24:47.959899Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:24:47.994231Z error port forward failed: lost connection to pod 2025-06-24T10:24:47.994511Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:24:48.223193Z warn pod: echo1/sotw-v1-65f898ccd5-pt4z4 port forward terminated 2025-06-24T10:24:48.262151Z warn pod: echo1/sotw-v1-65f898ccd5-pt4z4 port forward reconnect success 2025-06-24T10:24:48.433702Z warn pod: echo1/b-v1-7bcb76fc6f-29ngt port forward terminated 2025-06-24T10:24:48.465127Z warn pod: echo1/b-v1-7bcb76fc6f-29ngt port forward reconnect success 2025-06-24T10:24:48.616263Z error port forward failed: lost connection to pod 2025-06-24T10:24:48.815694Z error port forward failed: lost connection to pod 2025-06-24T10:24:48.977188Z warn pod: echo1/server-v1-86cfb5b889-csx2d port forward terminated 2025-06-24T10:24:49.017709Z warn pod: echo1/server-v1-86cfb5b889-csx2d port forward reconnect success 2025-06-24T10:24:49.134306Z error port forward failed: lost connection to pod 2025-06-24T10:24:49.279037Z warn pod: echo1/a-v1-78f49b749b-trkfh port forward terminated 2025-06-24T10:24:49.305443Z warn pod: echo1/headless-v1-64d65d6c8b-t549v port forward terminated 2025-06-24T10:24:49.311197Z warn pod: echo1/a-v1-78f49b749b-trkfh port forward reconnect success 2025-06-24T10:24:49.336304Z warn pod: echo1/headless-v1-64d65d6c8b-t549v port forward reconnect success 2025-06-24T10:24:49.562596Z error port forward failed: lost connection to pod 2025-06-24T10:24:50.028499Z warn pod: echo1/c-v1-6975698f5d-gp279 port forward terminated 2025-06-24T10:24:50.067687Z warn pod: echo1/c-v1-6975698f5d-gp279 port forward reconnect success 2025-06-24T10:24:50.154509Z error port forward failed: lost connection to pod 2025-06-24T10:24:50.319988Z warn pod: echo1/client-v1-546c55dc9c-8ddkj port forward terminated 2025-06-24T10:24:50.354763Z warn pod: echo1/client-v1-546c55dc9c-8ddkj port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.72:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.72:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:25:17.357221Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000982736s)' === 2025-06-24T10:25:17.357336Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:25:17.357396Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:25:17.357362Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:25:17.581705Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10128d64634-ba83-437a-98af-3cb33d1ac4e6 2025-06-24T10:25:17.581960Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2242s 2025-06-24T10:25:17.582698Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:17.680443Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1011ed2e4f7-6b67-4c9b-9778-cff8c8e3972e 2025-06-24T10:25:17.680731Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3229s 2025-06-24T10:25:17.681622Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:17.796578Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101178ab1c4-a9ec-4375-a027-21b96ef319e9 2025-06-24T10:25:17.796783Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4390s 2025-06-24T10:25:17.797557Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (re-run 2) (95.41s) 2025-06-24T10:23:43.898553Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:25:18.295056Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m34.396495187s)' === 2025-06-24T10:25:18.295171Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:25:18.295171Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:25:18.295183Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:25:18.463440Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/101a0c57c52-f941-40e8-88a7-0378cd89bc6b 2025-06-24T10:25:18.463636Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.1682s 2025-06-24T10:25:18.464375Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:18.588282Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10136856fc4-8f7e-4c88-90f9-4ce62dc758a7 2025-06-24T10:25:18.588623Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2932s 2025-06-24T10:25:18.589163Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:18.678039Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1010db2015e-68c6-4173-9723-ed8d4d526879 2025-06-24T10:25:18.678218Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3828s 2025-06-24T10:25:18.678843Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (re-run 2) (96.45s) 2025-06-24T10:23:43.875571Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:25:19.305607Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m35.430024915s)' === 2025-06-24T10:25:19.305718Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:25:19.305765Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:25:19.305729Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:25:19.478284Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/10117425831-8834-4ac4-a565-80081f03afec 2025-06-24T10:25:19.478642Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1726s 2025-06-24T10:25:19.479290Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:19.583783Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1010ec3d34c-ff74-45a3-9258-833d71432271 2025-06-24T10:25:19.583965Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2779s 2025-06-24T10:25:19.584703Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:25:19.665574Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:23:43Z/1012137a21e-7726-45f5-ac94-5c9583eeb7a1 2025-06-24T10:25:19.665719Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3596s 2025-06-24T10:25:19.666454Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 2) (31.25s) 2025-06-24T10:26:17.339309Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-06-24T10:26:47.340871Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.00155542s)' === 2025-06-24T10:26:47.340985Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:26:47.340969Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:26:47.341064Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-06-24T10:26:47.730416Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11814f90281-2eae-49d1-b0ec-15695556187a 2025-06-24T10:26:47.730728Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3894s 2025-06-24T10:26:47.731473Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:47.834463Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118498fa37f-0c80-437e-88da-314149243a3d 2025-06-24T10:26:47.834670Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4933s 2025-06-24T10:26:47.835536Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:26:47.990584Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11869ac6208-cf93-4ed2-a89e-c7668f19ce72 2025-06-24T10:26:47.990906Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6496s 2025-06-24T10:26:47.991275Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (re-run 2) (30.90s) 2025-06-24T10:26:48.591243Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-06-24T10:27:06.384590Z error port forward failed: lost connection to pod 2025-06-24T10:27:06.678581Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-t8tmt port forward terminated 2025-06-24T10:27:06.716313Z warn pod: echo1/server-naked-bar-v1-fdfcb6f6c-t8tmt port forward reconnect success 2025-06-24T10:27:06.974761Z error port forward failed: lost connection to pod 2025-06-24T10:27:07.043271Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-qvxzg port forward terminated 2025-06-24T10:27:07.046373Z error port forward failed: lost connection to pod 2025-06-24T10:27:07.081153Z warn pod: echo1/server-naked-foo-v1-6887fb7d79-qvxzg port forward reconnect success 2025-06-24T10:27:07.165460Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-z9694 port forward terminated 2025-06-24T10:27:07.210050Z warn pod: echo1/server-naked-foo-alt-v1-6b6b79fbc9-z9694 port forward reconnect success 2025-06-24T10:27:08.068936Z error port forward failed: lost connection to pod 2025-06-24T10:27:08.282774Z warn pod: external/external-v1-f9d74566f-x8q94 port forward terminated 2025-06-24T10:27:08.316018Z warn pod: external/external-v1-f9d74566f-x8q94 port forward reconnect success 2025-06-24T10:27:08.838172Z error port forward failed: lost connection to pod 2025-06-24T10:27:09.229256Z warn pod: echo1/proxyless-grpc-v1-8996df688-hpxhh port forward terminated 2025-06-24T10:27:09.261507Z warn pod: echo1/proxyless-grpc-v1-8996df688-hpxhh port forward reconnect success 2025-06-24T10:27:18.480015Z error port forward failed: lost connection to pod test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:27:18.592762Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.001513868s)' === 2025-06-24T10:27:18.592882Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:27:18.592886Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:27:18.592931Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-06-24T10:27:18.827593Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118ed67cda2-26b3-42f4-bc07-5427550d4cde 2025-06-24T10:27:18.827960Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2347s 2025-06-24T10:27:18.828571Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:18.917343Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1189bf3dee4-047d-4b76-9420-b9ebfce78e19 2025-06-24T10:27:18.917544Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3242s 2025-06-24T10:27:18.918194Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:18.950523Z warn pod: echo1/sotw-v1-65f898ccd5-2lp6d port forward terminated 2025-06-24T10:27:19.007046Z warn pod: echo1/sotw-v1-65f898ccd5-2lp6d port forward reconnect success 2025-06-24T10:27:19.013490Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1180c219b1c-2563-4067-94c2-75c9cdd6d53a 2025-06-24T10:27:19.013680Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4203s 2025-06-24T10:27:19.014161Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 2) (30.80s) 2025-06-24T10:27:19.488257Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-06-24T10:27:19.666199Z error port forward failed: lost connection to pod 2025-06-24T10:27:19.692927Z error port forward failed: lost connection to pod 2025-06-24T10:27:19.727786Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-06-24T10:27:19.760839Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-06-24T10:27:19.906498Z error port forward failed: lost connection to pod 2025-06-24T10:27:20.049992Z warn pod: echo1/a-v1-78f49b749b-mp5k2 port forward terminated 2025-06-24T10:27:20.081360Z warn pod: echo1/a-v1-78f49b749b-mp5k2 port forward reconnect success 2025-06-24T10:27:20.165676Z warn pod: echo1/headless-v1-64d65d6c8b-76jtq port forward terminated 2025-06-24T10:27:20.198658Z warn pod: echo1/headless-v1-64d65d6c8b-76jtq port forward reconnect success 2025-06-24T10:27:20.543999Z error port forward failed: lost connection to pod 2025-06-24T10:27:20.722624Z error port forward failed: lost connection to pod 2025-06-24T10:27:20.812698Z warn pod: echo1/server-v1-86cfb5b889-r86pq port forward terminated 2025-06-24T10:27:20.841229Z warn pod: echo1/server-v1-86cfb5b889-r86pq port forward reconnect success 2025-06-24T10:27:21.200464Z warn pod: echo1/b-v1-7bcb76fc6f-ghzdk port forward terminated 2025-06-24T10:27:21.232697Z warn pod: echo1/b-v1-7bcb76fc6f-ghzdk port forward reconnect success 2025-06-24T10:27:21.430231Z error port forward failed: lost connection to pod 2025-06-24T10:27:21.675373Z warn pod: echo1/c-v1-6975698f5d-s8mwt port forward terminated 2025-06-24T10:27:21.712123Z warn pod: echo1/c-v1-6975698f5d-s8mwt port forward reconnect success 2025-06-24T10:27:22.882120Z error port forward failed: lost connection to pod 2025-06-24T10:27:23.009001Z warn pod: echo1/client-v1-546c55dc9c-spwlj port forward terminated 2025-06-24T10:27:23.041485Z warn pod: echo1/client-v1-546c55dc9c-spwlj port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.79:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.79:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-06-24T10:27:49.488947Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000686659s)' === 2025-06-24T10:27:49.489038Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:27:49.489043Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:27:49.489091Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-06-24T10:27:49.643280Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11846e9a4b6-b197-4323-866f-3b881911b253 2025-06-24T10:27:49.643532Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1541s 2025-06-24T10:27:49.644114Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:49.730059Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118f7dc2eda-7e74-40bb-af2c-af5be7ee499b 2025-06-24T10:27:49.730385Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2409s 2025-06-24T10:27:49.731099Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:49.802340Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118210a2c66-7ad0-4b62-a2d2-4e532a240054 2025-06-24T10:27:49.802487Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3130s 2025-06-24T10:27:49.803082Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (re-run 2) (94.74s) 2025-06-24T10:26:16.465146Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-06-24T10:27:50.289877Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m33.824722936s)' === 2025-06-24T10:27:50.290021Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:27:50.290041Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:27:50.290064Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-06-24T10:27:50.455757Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1187ba782c6-fee8-40c3-a520-4d5b5b108f6f 2025-06-24T10:27:50.456140Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1659s 2025-06-24T10:27:50.456725Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:50.538201Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118ecd0aa94-dfb8-4bfc-8171-fb207827761e 2025-06-24T10:27:50.538418Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2481s 2025-06-24T10:27:50.539006Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:50.635334Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/1189831f22f-acc5-4ba7-b4ba-3757c87ce67c 2025-06-24T10:27:50.635555Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3452s 2025-06-24T10:27:50.636039Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (re-run 2) (95.77s) 2025-06-24T10:26:16.426659Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-06-24T10:27:51.205724Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m34.779051747s)' === 2025-06-24T10:27:51.205886Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-06-24T10:27:51.205900Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-06-24T10:27:51.205913Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-06-24T10:27:51.418792Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118434a4f51-aeab-4e2d-a67e-859e87cddea0 2025-06-24T10:27:51.419035Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2128s 2025-06-24T10:27:51.419824Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:51.529897Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/118492ab166-e4e1-4162-ac82-a38383798230 2025-06-24T10:27:51.530215Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3240s 2025-06-24T10:27:51.530813Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-06-24T10:27:51.618302Z info adsc Received type=istio.io/debug count=1 nonce=2025-06-24T10:26:16Z/11874b632f4-1163-4bb2-9215-f5cb01dc56d8 2025-06-24T10:27:51.618518Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4123s 2025-06-24T10:27:51.619084Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 3 runs, 63 tests, 31 failures in 892.294s exec: go version