exec: [go test -json -tags=integ -timeout 180m ./tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3046646 2025-08-04T18:04:38.670978Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:04:38.671032Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:04:38.715349Z info tf === Test Framework Settings === 2025-08-04T18:04:38.715494Z info tf TestID: security_ca_custom_root RunID: 452d8f4f-b4a6-404d-8448-50be57dd3811 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T18:04:38.715501Z info tf =============================== 2025-08-04T18:04:38.720812Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-452d8f4 2025-08-04T18:04:38.720933Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T18:04:38.720952Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T18:04:38.720975Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T18:04:38.720986Z info tf === BEGIN: Building clusters === 2025-08-04T18:04:38.728352Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T18:04:38.728400Z info tf === DONE: Building clusters === 2025-08-04T18:04:38.728431Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T18:04:39.298666Z info configmap istio-ca-root-cert is deleted 2025-08-04T18:04:39.298700Z info tf === SUCCESS: context setup function === 2025-08-04T18:04:39.298707Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T18:04:39.298725Z info tf === Istio Component Config === 2025-08-04T18:04:39.298776Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T18:04:39.298782Z info tf ================================ 2025-08-04T18:04:39.301678Z info tf skipping deployment as specified in the config 2025-08-04T18:04:39.301700Z info tf === SUCCEEDED: Deploy Istio in 2.98432ms [Suite=security_ca_custom_root]=== 2025-08-04T18:04:39.301712Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T18:04:39.967131Z info tf === SUCCEEDED: Create namespace echo1 in 665.401221ms === Generating RSA private key, 2048 bit long modulus (2 primes) .................................................................................................+++++ ....................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-452d8f4/_suite_context/ca-custom-root-4165382379/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ..............................................+++++ ..........................................................................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-452d8f4/_suite_context/ca-custom-root-4165382379/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ...................+++++ ............................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-452d8f4/_suite_context/ca-custom-root-4165382379/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T18:04:40.865210Z info tf === BEGIN: Create namespace external === 2025-08-04T18:04:41.929260Z info tf === SUCCEEDED: Create namespace external in 1.064060259s === 2025-08-04T18:04:45.439682Z info tf === BEGIN: Deploy echo instances === 2025-08-04T18:04:49.024219Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.038786Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.050167Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.062348Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.071642Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.085805Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.089074Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.093003Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.100572Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.100792Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.108490Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.112893Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.119660Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.124405Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.127841Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:04:49.139708Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.150546Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.163236Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.172868Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.186474Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.190011Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.193393Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.200754Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.200892Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.209314Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.213776Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.220176Z info klog Caches are synced controller=echo 2025-08-04T18:04:49.228863Z info klog Caches are synced controller=echo 2025-08-04T18:04:58.720236Z info tf === SUCCEEDED: Deploy echo instances in 13.280551615s === 2025-08-04T18:04:58.754636Z info tf === DONE: Setup: 'security_ca_custom_root' (20.026189144s) === 2025-08-04T18:04:58.754692Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === PASS tests/integration/security/ca_custom_root.TestMultiRootSetup/From_primary-0/server:server-naked-foo-alt[success] (1.03s) PASS tests/integration/security/ca_custom_root.TestMultiRootSetup/From_primary-0 (1.03s) PASS tests/integration/security/ca_custom_root.TestMultiRootSetup (1.07s) === RUN TestSecureNaming 2025-08-04T18:04:59.826790Z info tf === BEGIN: Test: 'security_ca_custom_root[TestSecureNaming]' === secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:114: retry.UntilSuccessOrFail: timeout while waiting after 10 attempts (last error: CA certificate (-----BEGIN CERTIFICATE----- MIIC/DCCAeSgAwIBAgIQTUjJ+mZ0yr7AX2hpmhvyijANBgkqhkiG9w0BAQsFADAY MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTI1MDgwNDE4MDMyNFoXDTM1MDgw MjE4MDMyNFowGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBAN3CS/Jf1TezHvDFoIQ6+C4//pRvjHH3o3Uh3GVJ fg2YtrHmj4iq9Eq9zloergCXnnNcqFagVdynEuqurTMFF1gD8kbKq+8ozYaD18M/ 9f++FR8WBstpC4HT1TSd/YSGrjCfuiU37xdJEeJG6QkzQhz4J+7bVR8o/6CSVH7Z NDsqrolQ09+RyvBlDFUemVm+OZ7ZeC3je7b/pKl5Uscf6mbKR52iQdAYXACKONrk mj3xk7d0j66nzXRjWOSSS5Is5ULRH22P2cqkyk6QNdxHxg4sn0oyVVhM9q4QWXQH WlWElJxnMWRTkkVk4OTg0XKFLM1ZKKgZFWQ2zkypGBUbjNkCAwEAAaNCMEAwDgYD VR0PAQH/BAQDAgIEMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNnL/E0WEoZP XFVnyiqtL1HuWxyAMA0GCSqGSIb3DQEBCwUAA4IBAQCkPML6p6LFIIddOUsRAlCG GcZPPkpbViSer27UsOG/4HEWzp3irPfLXQZI7Aoa26NYYqwnLHya5WSrER6joffa 7jiApP9hFND3PLOQdoe74VCVep+Q2orjmv3kKRojvYn3KrJp8vXzdk71tjYMUXo0 x6V2toTPupClFajehMoOA6PBQ/0jjHMdj5szXndBlPD0g9duWx4I7pY9x2ZOCr3u 5KOmx4CephIaURg6F70HMSrKlLHyEc8zKg1mNmNDCKktwJ6ki495+6tUQ9LaQ+Wh iZBSq2nKF55D4hTvMbIxcALMb3Zwjw4HAvBmQm28wuxLhf4VljgZhZQ0UOnWntfS -----END CERTIFICATE----- ) not matching plugin cert (-----BEGIN CERTIFICATE----- MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN 8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a 3g== -----END CERTIFICATE----- )) 2025-08-04T18:05:09.835104Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestSecureNaming] (10.008306871s)' === 2025-08-04T18:05:09.835226Z error tf === Dumping Namespace external State for TestSecureNaming... 2025-08-04T18:05:09.835240Z error tf === Dumping Namespace echo1 State for TestSecureNaming... 2025-08-04T18:05:09.835226Z error tf === Dumping Istio Deployment State for TestSecureNaming... 2025-08-04T18:05:09.881699Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:09.945647Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:10.060916Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:10.111689Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:10.175100Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:04:59Z/16a0f063ea-af71-4582-9599-4fd03cae85f7 2025-08-04T18:05:10.175268Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3396s 2025-08-04T18:05:10.175886Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:10.330102Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:04:59Z/16edcbbdb9-3e20-494e-8f93-ff1806c0f41b 2025-08-04T18:05:10.330344Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4947s 2025-08-04T18:05:10.330746Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:10.460325Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:04:59Z/16caccc4ff-99af-4e35-a73a-9996349845c5 2025-08-04T18:05:10.460447Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6248s 2025-08-04T18:05:10.460861Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:10.796680Z error tf === Dumping Namespace istio-system State for TestSecureNaming... 2025-08-04T18:05:10.807975Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:10.816942Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:10.886987Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:10.889164Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) --- FAIL: TestSecureNaming (11.08s) FAIL tests/integration/security/ca_custom_root.TestSecureNaming (11.08s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] 2025-08-04T18:05:10.933861Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.23.1.237:39386->172.30.189.239:443: read: connection reset by peer 2025-08-04T18:05:40.934361Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.000495869s)' === 2025-08-04T18:05:40.934461Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T18:05:40.934478Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T18:05:40.934470Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T18:05:41.000386Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:41.059199Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:41.236540Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:41.236554Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:41.310654Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17e282a81d-b220-4714-adb8-91a2429b5659 2025-08-04T18:05:41.310876Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3759s 2025-08-04T18:05:41.311498Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:41.476479Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/178852609f-bc5e-4635-9a28-58495e7aea03 2025-08-04T18:05:41.476600Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5416s 2025-08-04T18:05:41.477220Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:41.621924Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/173c290d34-fe09-43fe-9a55-e9707af456e6 2025-08-04T18:05:41.622026Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6870s 2025-08-04T18:05:41.622830Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (30.97s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (30.97s) PASS tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-bar[fail] (0.03s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0 2025-08-04T18:05:10.933594Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T18:05:41.932888Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (30.999294289s)' === 2025-08-04T18:05:41.932936Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T18:05:41.932963Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T18:05:41.932963Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T18:05:41.960638Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:42.012476Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:42.155758Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:42.155885Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:42.260325Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17aad80cc9-45a2-4654-b398-8fe917b54873 2025-08-04T18:05:42.260590Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3273s 2025-08-04T18:05:42.262219Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:42.417949Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/1748acd39a-0bd0-49f9-87b1-1ab8621574c5 2025-08-04T18:05:42.418052Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4847s 2025-08-04T18:05:42.418799Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:42.553866Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17e96147b8-eec4-49b7-b314-060e1ab60fec 2025-08-04T18:05:42.553989Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6206s 2025-08-04T18:05:42.554690Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0 (31.98s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0 (31.98s) === RUN TestTrustDomainAliasSecureNaming 2025-08-04T18:05:10.902987Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T18:05:42.918118Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.015122788s)' === 2025-08-04T18:05:42.918241Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T18:05:42.918241Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T18:05:42.918257Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T18:05:43.053491Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:43.185825Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:43.392390Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:43.433119Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:43.511464Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17c37e47a9-1e2e-4603-b04d-25a58e60ff87 2025-08-04T18:05:43.511790Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5934s 2025-08-04T18:05:43.512321Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:44.085091Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17043b7737-500d-4696-ab89-f68f74134d45 2025-08-04T18:05:44.085226Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1668s 2025-08-04T18:05:44.086014Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:44.525352Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/1766f657f3-ad86-4d33-a7b0-2cefb08bc451 2025-08-04T18:05:44.525519Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.6070s 2025-08-04T18:05:44.526073Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainAliasSecureNaming (34.53s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming (34.53s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:http-plaintext[allow] (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:tcp-plaintext[allow] (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:http-plaintext[allow] (0.03s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:tcp-plaintext[allow] (0.01s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:http-mtls[allow] (0.03s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:tcp-mtls[allow] (0.02s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:http-mtls[deny] (0.69s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls[deny] (0.03s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls-pass-through[deny] (0.03s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-08-04T18:05:46.580448Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === 2025-08-04T18:05:58.783412Z error port forward failed: lost connection to pod 2025-08-04T18:05:58.814887Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward terminated 2025-08-04T18:05:58.838276Z error port forward failed: lost connection to pod 2025-08-04T18:05:58.870493Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward terminated 2025-08-04T18:05:58.879708Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward reconnect success 2025-08-04T18:05:58.941785Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward reconnect success 2025-08-04T18:05:59.155648Z error port forward failed: lost connection to pod 2025-08-04T18:05:59.230288Z error port forward failed: lost connection to pod 2025-08-04T18:05:59.369553Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward terminated 2025-08-04T18:05:59.431943Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward reconnect success 2025-08-04T18:05:59.441407Z warn pod: external/external-v1-987785fb7-78rs2 port forward terminated 2025-08-04T18:05:59.503990Z warn pod: external/external-v1-987785fb7-78rs2 port forward reconnect success 2025-08-04T18:06:01.246717Z error port forward failed: lost connection to pod 2025-08-04T18:06:01.392945Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward terminated 2025-08-04T18:06:01.490666Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward reconnect success 2025-08-04T18:06:02.057184Z error port forward failed: lost connection to pod 2025-08-04T18:06:02.174510Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward terminated 2025-08-04T18:06:02.232712Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward reconnect success 2025-08-04T18:06:02.879757Z error port forward failed: lost connection to pod 2025-08-04T18:06:02.966306Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward terminated 2025-08-04T18:06:03.051682Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward reconnect success 2025-08-04T18:06:03.059040Z error port forward failed: lost connection to pod 2025-08-04T18:06:03.088345Z error port forward failed: lost connection to pod 2025-08-04T18:06:03.230854Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward terminated 2025-08-04T18:06:03.289302Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward reconnect success 2025-08-04T18:06:03.477416Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-08-04T18:06:03.537657Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-08-04T18:06:03.874399Z error port forward failed: lost connection to pod 2025-08-04T18:06:04.029392Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward terminated 2025-08-04T18:06:04.126109Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward reconnect success 2025-08-04T18:06:04.163826Z error port forward failed: lost connection to pod 2025-08-04T18:06:04.164084Z error port forward failed: lost connection to pod 2025-08-04T18:06:04.371101Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward terminated 2025-08-04T18:06:04.425896Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward terminated 2025-08-04T18:06:04.431931Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward reconnect success 2025-08-04T18:06:04.483187Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T18:06:16.581577Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.001122914s)' === 2025-08-04T18:06:16.581665Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:06:16.581686Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:06:16.581688Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:06:16.637220Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:06:16.681500Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:06:16.834526Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:16.834555Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:16.912600Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20bc87a754-4076-47c0-8467-dec9b7fbc624 2025-08-04T18:06:16.912910Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3309s 2025-08-04T18:06:16.913520Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:17.184890Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20a190fd16-1d39-4ca7-81b8-bcbe9ae7fa6e 2025-08-04T18:06:17.184978Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6029s 2025-08-04T18:06:17.185532Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:17.570111Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20cf615e1b-62d4-43d1-8b26-9838155ccb57 2025-08-04T18:06:17.570245Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9882s 2025-08-04T18:06:17.570919Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (31.50s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (31.50s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] 2025-08-04T18:06:18.076857Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T18:06:48.078165Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.00129715s)' === 2025-08-04T18:06:48.078268Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T18:06:48.078306Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T18:06:48.078309Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T18:06:48.156639Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: image can't be pulled 2025-08-04T18:06:48.208069Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:06:48.364678Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:48.390363Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:48.406085Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20fe7fd8e8-4a47-4dc0-b02d-1d565cf90127 2025-08-04T18:06:48.406361Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3277s 2025-08-04T18:06:48.407148Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:48.549294Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20eb085a4e-9844-433d-89b2-59176c183d22 2025-08-04T18:06:48.549448Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4707s 2025-08-04T18:06:48.550087Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:48.830870Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20c288650c-a2b6-42fe-99b1-cb0cd112f0bd 2025-08-04T18:06:48.831021Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.7522s 2025-08-04T18:06:48.831630Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (31.29s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (31.29s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] 2025-08-04T18:06:49.367643Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-08-04T18:06:51.832983Z error port forward failed: lost connection to pod 2025-08-04T18:06:52.033322Z warn pod: echo1/client-v1-9cfd95bd4-dwxrh port forward terminated 2025-08-04T18:06:52.621434Z warn pod: echo1/client-v1-9cfd95bd4-dwxrh port forward reconnect success 2025-08-04T18:07:04.883745Z error port forward failed: lost connection to pod 2025-08-04T18:07:04.946154Z error port forward failed: lost connection to pod 2025-08-04T18:07:05.315277Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward terminated 2025-08-04T18:07:05.370441Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward terminated 2025-08-04T18:07:05.380078Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward reconnect success 2025-08-04T18:07:05.431566Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward reconnect success 2025-08-04T18:07:05.439004Z error port forward failed: lost connection to pod 2025-08-04T18:07:05.508086Z error port forward failed: lost connection to pod 2025-08-04T18:07:05.870593Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward terminated 2025-08-04T18:07:05.941279Z warn pod: external/external-v1-987785fb7-78rs2 port forward terminated 2025-08-04T18:07:05.958086Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward reconnect success 2025-08-04T18:07:06.016343Z warn pod: external/external-v1-987785fb7-78rs2 port forward reconnect success 2025-08-04T18:07:07.496012Z error port forward failed: lost connection to pod 2025-08-04T18:07:07.893541Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward terminated 2025-08-04T18:07:07.954786Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward reconnect success 2025-08-04T18:07:08.237374Z error port forward failed: lost connection to pod 2025-08-04T18:07:08.673931Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward terminated 2025-08-04T18:07:08.732446Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward reconnect success 2025-08-04T18:07:09.057065Z error port forward failed: lost connection to pod 2025-08-04T18:07:09.294235Z error port forward failed: lost connection to pod 2025-08-04T18:07:09.465408Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward terminated 2025-08-04T18:07:09.541543Z error port forward failed: lost connection to pod 2025-08-04T18:07:09.544625Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward reconnect success 2025-08-04T18:07:09.730312Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward terminated 2025-08-04T18:07:09.799023Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward reconnect success 2025-08-04T18:07:09.978020Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-08-04T18:07:10.044694Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-08-04T18:07:10.131438Z error port forward failed: lost connection to pod 2025-08-04T18:07:10.437279Z error port forward failed: lost connection to pod 2025-08-04T18:07:10.489797Z error port forward failed: lost connection to pod 2025-08-04T18:07:10.529056Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward terminated 2025-08-04T18:07:10.589129Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward reconnect success 2025-08-04T18:07:10.871186Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward terminated 2025-08-04T18:07:10.926460Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward terminated 2025-08-04T18:07:10.954609Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward reconnect success 2025-08-04T18:07:11.002504Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.203:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.203:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T18:07:19.368367Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000718998s)' === 2025-08-04T18:07:19.368468Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T18:07:19.368467Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T18:07:19.368488Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T18:07:19.838480Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:19.903970Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:20.193162Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:20.195028Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:20.230860Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/2033f67e26-dd71-4563-844f-17e4ee7b0844 2025-08-04T18:07:20.231104Z info tf istioctl ([x internal-debug --all configz]): completed after 0.8621s 2025-08-04T18:07:20.231781Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:20.633490Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/201ba1c2dd-1592-4b1c-b9bd-7ddfb457cc7c 2025-08-04T18:07:20.633747Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.2647s 2025-08-04T18:07:20.634338Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:20.767405Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20242d8594-874e-410b-b144-7b1f5de95f0d 2025-08-04T18:07:20.767549Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.3986s 2025-08-04T18:07:20.768338Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.90s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.90s) === RUN TestTrustDomainValidation/From_primary-0 2025-08-04T18:05:45.648659Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T18:07:21.265321Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m35.616652377s)' === 2025-08-04T18:07:21.265443Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:07:21.265455Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:07:21.265478Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:07:21.305371Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:21.394947Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:21.460320Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:21.491028Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:21.547229Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20096c9bff-56fc-43a3-9b6f-d6fe4f2c531c 2025-08-04T18:07:21.547374Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2817s 2025-08-04T18:07:21.548157Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:21.682008Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/2027b05975-8b65-43a3-a6b9-57fbb8cb8c78 2025-08-04T18:07:21.682165Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4165s 2025-08-04T18:07:21.682816Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:21.847583Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/202bdf121a-1378-4f79-9ae0-1fda0efe9aa7 2025-08-04T18:07:21.847835Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5822s 2025-08-04T18:07:21.848519Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (96.60s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (96.60s) === RUN TestTrustDomainValidation 2025-08-04T18:05:45.434860Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T18:07:22.246134Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m36.811259564s)' === 2025-08-04T18:07:22.246277Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T18:07:22.246276Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T18:07:22.246300Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T18:07:22.305501Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:22.335267Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:22.462860Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:22.472184Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:22.523950Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/208a325929-27ba-442f-95b8-a1390a8004ee 2025-08-04T18:07:22.524193Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2777s 2025-08-04T18:07:22.524642Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:22.660667Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/205ad94894-cc06-419d-9061-2eb2000b8c38 2025-08-04T18:07:22.660797Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4142s 2025-08-04T18:07:22.661382Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:22.823614Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20603c513a-8686-4df2-b250-fbcec06377c3 2025-08-04T18:07:22.823831Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5772s 2025-08-04T18:07:22.824481Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (97.84s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (97.84s) 2025-08-04T18:07:23.279081Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-08-04T18:07:23.279189Z info tf === Suite "security_ca_custom_root" run time: 2m44.550764451s === 2025-08-04T18:07:23.279214Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-08-04T18:07:23.281327Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:23.281328Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:23.281388Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:23.338757Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:23.452808Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:23.519173Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:23.532698Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:23.612571Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:07:23Z/2123301b89-37e7-4298-a7ce-8b2014e3173f 2025-08-04T18:07:23.612843Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3308s 2025-08-04T18:07:23.613521Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:23.763858Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:07:23Z/212e87a276-6106-4d1d-b4d3-05b5905464d1 2025-08-04T18:07:23.764052Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4820s 2025-08-04T18:07:23.764637Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:23.908985Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:07:23Z/2142dd9965-e0cc-4bcd-ad39-a2f5eb8d99e5 2025-08-04T18:07:23.909146Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6271s 2025-08-04T18:07:23.909747Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:24.379184Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T18:07:24.379329Z info tf === SUCCEEDED: Cleanup Istio in 131.047µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root DONE 22 tests, 9 failures in 165.741s exec: [go test -json -test.run=^TestSecureNaming$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3048831 2025-08-04T18:07:40.455615Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:07:40.455658Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:07:40.486830Z info tf === Test Framework Settings === 2025-08-04T18:07:40.486925Z info tf TestID: security_ca_custom_root RunID: 1c4dd152-3e17-415e-88c7-fe357220b92d NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T18:07:40.486932Z info tf =============================== 2025-08-04T18:07:40.487074Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15 2025-08-04T18:07:40.487170Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T18:07:40.487183Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T18:07:40.487201Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T18:07:40.487209Z info tf === BEGIN: Building clusters === 2025-08-04T18:07:40.491061Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T18:07:40.491069Z info tf === DONE: Building clusters === 2025-08-04T18:07:40.491086Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T18:07:40.545651Z info configmap istio-ca-root-cert is deleted 2025-08-04T18:07:40.545707Z info tf === SUCCESS: context setup function === 2025-08-04T18:07:40.545716Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T18:07:40.545735Z info tf === Istio Component Config === 2025-08-04T18:07:40.545790Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T18:07:40.545799Z info tf ================================ 2025-08-04T18:07:40.548956Z info tf skipping deployment as specified in the config 2025-08-04T18:07:40.548975Z info tf === SUCCEEDED: Deploy Istio in 3.253455ms [Suite=security_ca_custom_root]=== 2025-08-04T18:07:40.548984Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T18:07:40.560880Z info tf === SUCCEEDED: Create namespace echo1 in 11.881048ms === Generating RSA private key, 2048 bit long modulus (2 primes) .......................................................+++++ ................................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15/_suite_context/ca-custom-root-3193693405/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ......+++++ .........+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15/_suite_context/ca-custom-root-3193693405/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ................................+++++ ..........................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15/_suite_context/ca-custom-root-3193693405/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T18:07:40.938406Z info tf === BEGIN: Create namespace external === 2025-08-04T18:07:40.948214Z info tf === SUCCEEDED: Create namespace external in 9.830269ms === 2025-08-04T18:07:41.039223Z info tf === BEGIN: Deploy echo instances === 2025-08-04T18:07:41.337107Z error tf === FAILED: Deploy echo instances === 2025-08-04T18:07:41.337176Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15/_suite_context/Service.628927797.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T18:07:41.337227Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:41.337233Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:41.337263Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:41.379203Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:41.418060Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:41.477412Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T18:07:41.477723Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:41.504256Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:41.544128Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:07:39Z/24d05affd0-0de3-4589-b989-f10e3cfbaf53 2025-08-04T18:07:41.544247Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2068s 2025-08-04T18:07:41.544808Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:41.680763Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:07:39Z/24aa9fdfeb-9c41-4ee4-910f-834efde37164 2025-08-04T18:07:41.680975Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3435s 2025-08-04T18:07:41.681431Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:41.814772Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15/_suite_context/Service.628927797.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T18:07:41.814855Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15/_suite_context/Service.628927797.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-08-04T18:07:41.814861Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-1c4dd15/_suite_context/Service.628927797.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T18:07:41.814890Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:41.814902Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:41.814911Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:41.828877Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:07:39Z/24d656b628-4685-404f-a9b7-2bee1e43ebcc 2025-08-04T18:07:41.830476Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4929s 2025-08-04T18:07:41.831233Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:41.838315Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:41.863239Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:41.943477Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "external-v1-987785fb7-78rs2" not found in the "external" namespace) 2025-08-04T18:07:42.005590Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:42.005928Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:42.047899Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:07:39Z/241d70bb33-8b37-4894-a988-e61e2e28d4b5 2025-08-04T18:07:42.048084Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2325s 2025-08-04T18:07:42.048620Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:42.338938Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T18:07:42.338993Z info tf === SUCCEEDED: Cleanup Istio in 49.674µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0$/^server:server-naked-foo\[success\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3050179 2025-08-04T18:07:55.257184Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:07:55.257238Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:07:55.284992Z info tf === Test Framework Settings === 2025-08-04T18:07:55.286243Z info tf TestID: security_ca_custom_root RunID: eb2f2db4-644d-4178-bd0b-d2e75c0f0258 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T18:07:55.286478Z info tf =============================== 2025-08-04T18:07:55.287224Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db 2025-08-04T18:07:55.287344Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T18:07:55.287361Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T18:07:55.287383Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T18:07:55.287394Z info tf === BEGIN: Building clusters === 2025-08-04T18:07:55.291787Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T18:07:55.291796Z info tf === DONE: Building clusters === 2025-08-04T18:07:55.291817Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T18:07:55.507433Z info configmap istio-ca-root-cert is deleted 2025-08-04T18:07:55.507486Z info tf === SUCCESS: context setup function === 2025-08-04T18:07:55.507494Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T18:07:55.507510Z info tf === Istio Component Config === 2025-08-04T18:07:55.507545Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T18:07:55.507550Z info tf ================================ 2025-08-04T18:07:55.510291Z info tf skipping deployment as specified in the config 2025-08-04T18:07:55.510304Z info tf === SUCCEEDED: Deploy Istio in 2.806741ms [Suite=security_ca_custom_root]=== 2025-08-04T18:07:55.510310Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T18:07:55.551530Z info tf === SUCCEEDED: Create namespace echo1 in 41.212315ms === Generating RSA private key, 2048 bit long modulus (2 primes) ............+++++ .......................................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db/_suite_context/ca-custom-root-1395140531/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ...+++++ ....+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db/_suite_context/ca-custom-root-1395140531/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ...........................................+++++ ....................................................................................................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db/_suite_context/ca-custom-root-1395140531/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T18:07:55.976854Z info tf === BEGIN: Create namespace external === 2025-08-04T18:07:56.019979Z info tf === SUCCEEDED: Create namespace external in 43.157425ms === 2025-08-04T18:07:56.292137Z info tf === BEGIN: Deploy echo instances === 2025-08-04T18:07:56.372764Z error tf === FAILED: Deploy echo instances === 2025-08-04T18:07:56.372799Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db/_suite_context/ServiceAccount-Service.3416857511.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:07:56.372855Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:56.372885Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:56.372925Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:56.427541Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db/_suite_context/ServiceAccount-Service.3416857511.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:07:56.427586Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db/_suite_context/ServiceAccount-Service.3416857511.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T18:07:56.427592Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-eb2f2db/_suite_context/ServiceAccount-Service.3416857511.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:07:56.427613Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:56.427630Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:56.427653Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:07:56.478711Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:56.548103Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:56.553583Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T18:07:56.553657Z info tf === SUCCEEDED: Cleanup Istio in 67.611µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$/^naked\[foo\]->server:http-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3051636 2025-08-04T18:08:10.059237Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:08:10.059301Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:08:10.090469Z info tf === Test Framework Settings === 2025-08-04T18:08:10.090588Z info tf TestID: security_ca_custom_root RunID: 6514b3f9-dbc8-47a2-9197-8491f76ca376 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T18:08:10.090594Z info tf =============================== 2025-08-04T18:08:10.090766Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6514b3f 2025-08-04T18:08:10.090856Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T18:08:10.090871Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T18:08:10.090891Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T18:08:10.090901Z info tf === BEGIN: Building clusters === 2025-08-04T18:08:10.094998Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T18:08:10.095007Z info tf === DONE: Building clusters === 2025-08-04T18:08:10.095029Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T18:08:10.304739Z info configmap istio-ca-root-cert is deleted 2025-08-04T18:08:10.304849Z info tf === SUCCESS: context setup function === 2025-08-04T18:08:10.304858Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T18:08:10.304874Z info tf === Istio Component Config === 2025-08-04T18:08:10.304926Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T18:08:10.304932Z info tf ================================ 2025-08-04T18:08:10.307982Z info tf skipping deployment as specified in the config 2025-08-04T18:08:10.308001Z info tf === SUCCEEDED: Deploy Istio in 3.136683ms [Suite=security_ca_custom_root]=== 2025-08-04T18:08:10.308008Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T18:08:10.452094Z info tf === SUCCEEDED: Create namespace echo1 in 144.061425ms === Generating RSA private key, 2048 bit long modulus (2 primes) ............................................................................................................+++++ ................................................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6514b3f/_suite_context/ca-custom-root-3043256308/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ..........................................+++++ .................................................................................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6514b3f/_suite_context/ca-custom-root-3043256308/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ........................+++++ ..............................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-6514b3f/_suite_context/ca-custom-root-3043256308/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T18:08:11.473844Z info tf === BEGIN: Create namespace external === 2025-08-04T18:08:11.846207Z info tf === SUCCEEDED: Create namespace external in 372.347168ms === 2025-08-04T18:08:13.925668Z info tf === BEGIN: Deploy echo instances === 2025-08-04T18:08:19.914261Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.015453Z info klog Caches are synced controller=echo 2025-08-04T18:08:20.317174Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.359515Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.418343Z info klog Caches are synced controller=echo 2025-08-04T18:08:20.460011Z info klog Caches are synced controller=echo 2025-08-04T18:08:20.470130Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.470131Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.570760Z info klog Caches are synced controller=echo 2025-08-04T18:08:20.570760Z info klog Caches are synced controller=echo 2025-08-04T18:08:20.781859Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.830419Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.882895Z info klog Caches are synced controller=echo 2025-08-04T18:08:20.925965Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:20.931243Z info klog Caches are synced controller=echo 2025-08-04T18:08:21.026840Z info klog Caches are synced controller=echo 2025-08-04T18:08:21.084557Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:21.185102Z info klog Caches are synced controller=echo 2025-08-04T18:08:21.252389Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:21.252393Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:21.277262Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:21.353035Z info klog Caches are synced controller=echo 2025-08-04T18:08:21.353035Z info klog Caches are synced controller=echo 2025-08-04T18:08:21.377942Z info klog Caches are synced controller=echo 2025-08-04T18:08:21.506122Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:21.526865Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:08:21.607271Z info klog Caches are synced controller=echo 2025-08-04T18:08:21.627812Z info klog Caches are synced controller=echo 2025-08-04T18:08:31.677670Z info tf === SUCCEEDED: Deploy echo instances in 17.751985482s === 2025-08-04T18:08:31.715787Z info tf === DONE: Setup: 'security_ca_custom_root' (21.620733272s) === 2025-08-04T18:08:31.715868Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-08-04T18:08:31.747100Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T18:09:01.747576Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000472033s)' === 2025-08-04T18:09:01.747669Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:09:01.747686Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:09:01.747696Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:09:01.987644Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:02.033547Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:02.328922Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:02.915766Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:03.008708Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4991db33cc-f800-4fc5-9baf-af2405bd0cc2 2025-08-04T18:09:03.008971Z info tf istioctl ([x internal-debug --all configz]): completed after 1.2609s 2025-08-04T18:09:03.009567Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:03.477437Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/49692d828b-790e-4cc9-8d22-554f92780d5d 2025-08-04T18:09:03.477611Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.7295s 2025-08-04T18:09:03.478247Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:03.603458Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/49a4c02fd2-c3cb-4dc0-a0e1-ae1ac9e0149f 2025-08-04T18:09:03.603540Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.8554s 2025-08-04T18:09:03.604119Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (32.24s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 1) (32.24s) === RUN TestTrustDomainValidation/From_primary-0 2025-08-04T18:08:31.746415Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T18:09:03.985511Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (32.239089047s)' === 2025-08-04T18:09:03.985621Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:09:03.985644Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:09:03.985676Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:09:04.083746Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:04.188910Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:04.520822Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:04.578336Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:04.608329Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4938668aa3-1893-4440-8607-27e9b74b09a0 2025-08-04T18:09:04.608539Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5892s 2025-08-04T18:09:04.609182Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:04.851608Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4988df792d-151e-48c7-80ff-a46089ba9c7a 2025-08-04T18:09:04.851779Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8324s 2025-08-04T18:09:04.852308Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:04.972279Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4916e9157e-f961-4d28-a140-ba31e00941c3 2025-08-04T18:09:04.972446Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9530s 2025-08-04T18:09:04.972896Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (33.43s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (re-run 1) (33.43s) === RUN TestTrustDomainValidation 2025-08-04T18:08:31.716537Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T18:09:05.181129Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (33.464577859s)' === 2025-08-04T18:09:05.181261Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T18:09:05.181261Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T18:09:05.181290Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T18:09:05.203160Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:05.294066Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:05.418135Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:05.467479Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:05.856196Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/494705aba9-d35a-45e5-b638-67cd9fe08e98 2025-08-04T18:09:05.856447Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6749s 2025-08-04T18:09:05.857117Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:07.137077Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/493cac5504-bfe7-40c7-9c21-bec8706eedfc 2025-08-04T18:09:07.137211Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.9557s 2025-08-04T18:09:07.137864Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:07.844998Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/49577646a9-8a6e-4097-80c9-89a47a0d7ae4 2025-08-04T18:09:07.845124Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.6636s 2025-08-04T18:09:07.845675Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (37.32s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (re-run 1) (37.32s) 2025-08-04T18:09:09.033423Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-08-04T18:09:09.033485Z info tf === Suite "security_ca_custom_root" run time: 58.938459973s === 2025-08-04T18:09:09.033500Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-08-04T18:09:09.033828Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:09.033828Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:09.033846Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:09.185516Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:09.236219Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:09.295537Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:09.417198Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:09.446417Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:09Z/505d108cd3-056d-48f8-b6ea-c2017b858528 2025-08-04T18:09:09.446574Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4124s 2025-08-04T18:09:09.447418Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:09.656165Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:09Z/50693c1cc8-82aa-4bfc-bdb7-451c9ca6b15e 2025-08-04T18:09:09.656275Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6220s 2025-08-04T18:09:09.656865Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:09.828365Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:09Z/500ad6a323-f38c-44cd-a796-c1d8ee98292f 2025-08-04T18:09:09.828634Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7944s 2025-08-04T18:09:09.829088Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:10.221848Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T18:09:10.222004Z info tf === SUCCEEDED: Cleanup Istio in 146.074µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$/^naked\[foo\]->server:tcp-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3053237 2025-08-04T18:09:25.054680Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:09:25.054740Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:09:25.083384Z info tf === Test Framework Settings === 2025-08-04T18:09:25.083490Z info tf TestID: security_ca_custom_root RunID: cdc4c19a-1e5c-425f-afdb-54a91d3267fb NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T18:09:25.083496Z info tf =============================== 2025-08-04T18:09:25.083650Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19 2025-08-04T18:09:25.083735Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T18:09:25.083746Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T18:09:25.083763Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T18:09:25.083771Z info tf === BEGIN: Building clusters === 2025-08-04T18:09:25.089735Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T18:09:25.089792Z info tf === DONE: Building clusters === 2025-08-04T18:09:25.089812Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T18:09:25.179584Z info configmap istio-ca-root-cert is deleted 2025-08-04T18:09:25.179645Z info tf === SUCCESS: context setup function === 2025-08-04T18:09:25.179659Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T18:09:25.179690Z info tf === Istio Component Config === 2025-08-04T18:09:25.179743Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T18:09:25.179752Z info tf ================================ 2025-08-04T18:09:25.183385Z info tf skipping deployment as specified in the config 2025-08-04T18:09:25.183408Z info tf === SUCCEEDED: Deploy Istio in 3.745099ms [Suite=security_ca_custom_root]=== 2025-08-04T18:09:25.183416Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T18:09:25.205913Z info tf === SUCCEEDED: Create namespace echo1 in 22.489559ms === Generating RSA private key, 2048 bit long modulus (2 primes) ....................+++++ .....+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19/_suite_context/ca-custom-root-3703899923/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ........................................................................................+++++ .........................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19/_suite_context/ca-custom-root-3703899923/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ................................................................................................+++++ .................................................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19/_suite_context/ca-custom-root-3703899923/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T18:09:25.711379Z info tf === BEGIN: Create namespace external === 2025-08-04T18:09:25.731929Z info tf === SUCCEEDED: Create namespace external in 20.57093ms === 2025-08-04T18:09:25.867116Z info tf === BEGIN: Deploy echo instances === 2025-08-04T18:09:25.888064Z error tf === FAILED: Deploy echo instances === 2025-08-04T18:09:25.888102Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19/_suite_context/ServiceAccount-Service.1276805871.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:09:25.888134Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:25.888163Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:25.888166Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:26.058941Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:26.367071Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:26.447181Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:26.628856Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-5f56c57bb9-jnk7z_echo1") 2025-08-04T18:09:26.654368Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-6b586f9d47-s77fp_echo1") 2025-08-04T18:09:26.684075Z error klog an error occurred forwarding 34019 -> 15020: error forwarding port 15020 to pod 5adae51434f0901781c6c2cc5930aaf44d090a2ecbe0d2e48e5ffd36e462d07f, uid : port forward into network namespace "/var/run/netns/c888c730-d6d5-46f8-bf6b-7b5f1b56994a": failed to connect to localhost:15020 inside namespace 5adae51434f0901781c6c2cc5930aaf44d090a2ecbe0d2e48e5ffd36e462d07f: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:26.685991Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-5bccd7df8d-9dm5q for: Get "http://localhost:34019/debug/ndsz": EOF 2025-08-04T18:09:26.686025Z error port forward failed: lost connection to pod 2025-08-04T18:09:26.686847Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-5bccd7df8d-9dm5q for: Get "http://localhost:34019/stats/prometheus": dial tcp 127.0.0.1:34019: connect: connection refused 2025-08-04T18:09:26.701149Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-5f56c57bb9-jnk7z_echo1") 2025-08-04T18:09:26.724553Z error klog an error occurred forwarding 33589 -> 15000: error forwarding port 15000 to pod 5adae51434f0901781c6c2cc5930aaf44d090a2ecbe0d2e48e5ffd36e462d07f, uid : port forward into network namespace "/var/run/netns/c888c730-d6d5-46f8-bf6b-7b5f1b56994a": failed to connect to localhost:15000 inside namespace 5adae51434f0901781c6c2cc5930aaf44d090a2ecbe0d2e48e5ffd36e462d07f: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:26.726533Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-5bccd7df8d-9dm5q for: Get "http://localhost:33589/config_dump?include_eds=true": EOF 2025-08-04T18:09:26.726572Z error port forward failed: lost connection to pod 2025-08-04T18:09:26.727494Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-5bccd7df8d-9dm5q for: Get "http://localhost:33589/clusters": dial tcp 127.0.0.1:33589: connect: connection refused 2025-08-04T18:09:26.728383Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:26.730265Z error klog an error occurred forwarding 35195 -> 15020: error forwarding port 15020 to pod 736d6f3480664430d90d227d7a4f9c3c6ee084caa7deb541b3d146ad9613a9ea, uid : port forward into network namespace "/var/run/netns/ab848817-ddba-42e5-9703-ebf173dc92f6": failed to connect to localhost:15020 inside namespace 736d6f3480664430d90d227d7a4f9c3c6ee084caa7deb541b3d146ad9613a9ea: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:26.731817Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-6d77f6b7c4-qx788 for: Get "http://localhost:35195/debug/ndsz": EOF 2025-08-04T18:09:26.731814Z error port forward failed: lost connection to pod 2025-08-04T18:09:26.732428Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-6d77f6b7c4-qx788 for: Get "http://localhost:35195/stats/prometheus": dial tcp 127.0.0.1:35195: connect: connection refused 2025-08-04T18:09:26.739529Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-v1-9cfd95bd4-655hd_echo1") 2025-08-04T18:09:26.743277Z error klog an error occurred forwarding 45307 -> 15000: error forwarding port 15000 to pod 6f559d9e172855319510cd8f9edf53e99725805f33f0f1dd083609022729fabd, uid : port forward into network namespace "/var/run/netns/29a5e9ba-c765-4ee0-8a16-bc203ebd3c1e": failed to connect to localhost:15000 inside namespace 6f559d9e172855319510cd8f9edf53e99725805f33f0f1dd083609022729fabd: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:26.743651Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6f468cf89c-92ddz_echo1") 2025-08-04T18:09:26.745465Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-5455bcc7b9-j5mhr for: Get "http://localhost:45307/config_dump?include_eds=true": EOF 2025-08-04T18:09:26.745527Z error port forward failed: lost connection to pod 2025-08-04T18:09:26.746031Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-5455bcc7b9-j5mhr for: Get "http://localhost:45307/clusters": dial tcp 127.0.0.1:45307: connect: connection refused 2025-08-04T18:09:26.764543Z error klog an error occurred forwarding 39157 -> 15020: error forwarding port 15020 to pod 6f559d9e172855319510cd8f9edf53e99725805f33f0f1dd083609022729fabd, uid : port forward into network namespace "/var/run/netns/29a5e9ba-c765-4ee0-8a16-bc203ebd3c1e": failed to connect to localhost:15020 inside namespace 6f559d9e172855319510cd8f9edf53e99725805f33f0f1dd083609022729fabd: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:26.766352Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-5455bcc7b9-j5mhr for: Get "http://localhost:39157/debug/ndsz": EOF 2025-08-04T18:09:26.766371Z error port forward failed: lost connection to pod 2025-08-04T18:09:26.767245Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-5455bcc7b9-j5mhr for: Get "http://localhost:39157/stats/prometheus": dial tcp 127.0.0.1:39157: connect: connection refused 2025-08-04T18:09:26.795328Z error klog an error occurred forwarding 41111 -> 15020: error forwarding port 15020 to pod 76b18c2f4fb83fe118ea17d256c6113128915575ef647ac4e61075f288cbd8be, uid : port forward into network namespace "/var/run/netns/faf1540e-360c-475b-a9f6-7a0c80a394d9": failed to connect to localhost:15020 inside namespace 76b18c2f4fb83fe118ea17d256c6113128915575ef647ac4e61075f288cbd8be: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:26.797157Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-c494d6b8d-hj56j for: Get "http://localhost:41111/debug/ndsz": EOF 2025-08-04T18:09:26.797192Z error port forward failed: lost connection to pod 2025-08-04T18:09:26.797855Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-c494d6b8d-hj56j for: Get "http://localhost:41111/stats/prometheus": dial tcp 127.0.0.1:41111: connect: connection refused 2025-08-04T18:09:26.864154Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-6b586f9d47-s77fp_echo1") 2025-08-04T18:09:27.069138Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-v1-9cfd95bd4-655hd_echo1") 2025-08-04T18:09:27.372627Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-5f56c57bb9-jnk7z_echo1") 2025-08-04T18:09:27.383968Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:27Z/55ab05a986-53d1-4596-b3d7-e4fcafacc333 2025-08-04T18:09:27.384162Z info tf istioctl ([x internal-debug --all configz]): completed after 1.4957s 2025-08-04T18:09:27.384730Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:27.395378Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-v1-9cfd95bd4-655hd_echo1") 2025-08-04T18:09:27.420728Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-6b586f9d47-s77fp_echo1") 2025-08-04T18:09:27.517051Z error port forward failed: error upgrading connection: unable to upgrade connection: pod does not exist 2025-08-04T18:09:27.573932Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6f468cf89c-92ddz_echo1") 2025-08-04T18:09:27.574218Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-6b586f9d47-s77fp_echo1") 2025-08-04T18:09:27.611075Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/c-v1-5f56c57bb9-jnk7z/app for: pods "c-v1-5f56c57bb9-jnk7z" not found 2025-08-04T18:09:27.748703Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/c-v1-5f56c57bb9-jnk7z/istio-proxy for: pods "c-v1-5f56c57bb9-jnk7z" not found 2025-08-04T18:09:27.766455Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-v1-9cfd95bd4-655hd_echo1") 2025-08-04T18:09:27.807330Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/c-v1-5f56c57bb9-jnk7z/istio-validation for: pods "c-v1-5f56c57bb9-jnk7z" not found 2025-08-04T18:09:27.849803Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-9cfd95bd4-655hd/app for: pods "client-v1-9cfd95bd4-655hd" not found 2025-08-04T18:09:27.878670Z error port forward failed: error upgrading connection: unable to upgrade connection: pod does not exist 2025-08-04T18:09:27.939422Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-9cfd95bd4-655hd/istio-proxy for: pods "client-v1-9cfd95bd4-655hd" not found 2025-08-04T18:09:27.988921Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-6f468cf89c-92ddz_echo1") 2025-08-04T18:09:28.011152Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-9cfd95bd4-655hd/istio-validation for: pods "client-v1-9cfd95bd4-655hd" not found 2025-08-04T18:09:28.021920Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "c-v1-5f56c57bb9-jnk7z" not found in the "echo1" namespace) 2025-08-04T18:09:28.054797Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T18:09:28.102920Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:27Z/55bfbfce42-5512-44b7-b8ef-6967265fdb20 2025-08-04T18:09:28.103083Z info tf istioctl ([x internal-debug --all mcsz]): completed after 2.2146s 2025-08-04T18:09:28.103881Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:28.227873Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-5455bcc7b9-j5mhr/istio-validation for: the server could not find the requested resource ( pods/log headless-v1-5455bcc7b9-j5mhr) 2025-08-04T18:09:28.280687Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-bar-v1-7cf5cd474f-xkkdd/app for: pods "server-naked-bar-v1-7cf5cd474f-xkkdd" not found 2025-08-04T18:09:28.356662Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-alt-v1-6cbc84d88f-sbs6r/app for: pods "server-naked-foo-alt-v1-6cbc84d88f-sbs6r" not found 2025-08-04T18:09:28.376921Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-v1-6f468cf89c-92ddz/app for: pods "server-naked-foo-v1-6f468cf89c-92ddz" not found 2025-08-04T18:09:28.622672Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:27Z/55b9b9dd3c-ed06-4094-9a18-9f86c3cbe586 2025-08-04T18:09:28.622809Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.7343s 2025-08-04T18:09:28.623543Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:28.810363Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19/_suite_context/ServiceAccount-Service.1276805871.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:09:28.810394Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19/_suite_context/ServiceAccount-Service.1276805871.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T18:09:28.810401Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-cdc4c19/_suite_context/ServiceAccount-Service.1276805871.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:09:28.810431Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:28.810449Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:28.810478Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:28.928628Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:28.981227Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:29.078604Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:29.080104Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:29.095833Z error klog an error occurred forwarding 33943 -> 15020: error forwarding port 15020 to pod 76b18c2f4fb83fe118ea17d256c6113128915575ef647ac4e61075f288cbd8be, uid : port forward into network namespace "/var/run/netns/faf1540e-360c-475b-a9f6-7a0c80a394d9": failed to connect to localhost:15020 inside namespace 76b18c2f4fb83fe118ea17d256c6113128915575ef647ac4e61075f288cbd8be: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:29.098476Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-c494d6b8d-hj56j for: Get "http://localhost:33943/debug/ndsz": EOF 2025-08-04T18:09:29.098478Z error port forward failed: lost connection to pod 2025-08-04T18:09:29.101076Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-c494d6b8d-hj56j for: Get "http://localhost:33943/stats/prometheus": dial tcp 127.0.0.1:33943: connect: connection refused 2025-08-04T18:09:29.141230Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:27Z/5526bf154c-8abc-4bf9-b41a-20d091c5b4e7 2025-08-04T18:09:29.141478Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3308s 2025-08-04T18:09:29.141924Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:29.159207Z error klog an error occurred forwarding 46123 -> 15020: error forwarding port 15020 to pod 736d6f3480664430d90d227d7a4f9c3c6ee084caa7deb541b3d146ad9613a9ea, uid : port forward into network namespace "/var/run/netns/ab848817-ddba-42e5-9703-ebf173dc92f6": failed to connect to localhost:15020 inside namespace 736d6f3480664430d90d227d7a4f9c3c6ee084caa7deb541b3d146ad9613a9ea: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T18:09:29.161039Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-6d77f6b7c4-qx788 for: Get "http://localhost:46123/debug/ndsz": EOF 2025-08-04T18:09:29.161076Z error port forward failed: lost connection to pod 2025-08-04T18:09:29.161815Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-6d77f6b7c4-qx788 for: Get "http://localhost:46123/stats/prometheus": dial tcp 127.0.0.1:46123: connect: connection refused 2025-08-04T18:09:29.489579Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:27Z/553a9f7e84-f232-4940-ba7a-03579042c477 2025-08-04T18:09:29.489697Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6790s 2025-08-04T18:09:29.490361Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:29.510823Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T18:09:29.510868Z info tf === SUCCEEDED: Cleanup Istio in 41.067µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$/^naked\[foo\]->server:tcp-mtls-pass-through\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3054651 2025-08-04T18:09:40.122452Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:09:40.122505Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:09:40.151572Z info tf === Test Framework Settings === 2025-08-04T18:09:40.151657Z info tf TestID: security_ca_custom_root RunID: 8cb3b82a-02a1-4d47-a59e-4f5708aa1753 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T18:09:40.151662Z info tf =============================== 2025-08-04T18:09:40.152161Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82 2025-08-04T18:09:40.152271Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T18:09:40.152286Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T18:09:40.152309Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T18:09:40.152319Z info tf === BEGIN: Building clusters === 2025-08-04T18:09:40.156034Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T18:09:40.156042Z info tf === DONE: Building clusters === 2025-08-04T18:09:40.156055Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T18:09:40.439092Z info configmap istio-ca-root-cert is deleted 2025-08-04T18:09:40.439170Z info tf === SUCCESS: context setup function === 2025-08-04T18:09:40.439178Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T18:09:40.439200Z info tf === Istio Component Config === 2025-08-04T18:09:40.439249Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T18:09:40.439255Z info tf ================================ 2025-08-04T18:09:40.441928Z info tf skipping deployment as specified in the config 2025-08-04T18:09:40.441942Z info tf === SUCCEEDED: Deploy Istio in 2.761059ms [Suite=security_ca_custom_root]=== 2025-08-04T18:09:40.441949Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T18:09:40.501392Z info tf === SUCCEEDED: Create namespace echo1 in 59.435962ms === Generating RSA private key, 2048 bit long modulus (2 primes) ................................+++++ .....+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82/_suite_context/ca-custom-root-2491517925/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ...........................................+++++ ..............................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82/_suite_context/ca-custom-root-2491517925/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) .+++++ ....................................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82/_suite_context/ca-custom-root-2491517925/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T18:09:40.884240Z info tf === BEGIN: Create namespace external === 2025-08-04T18:09:40.898963Z info tf === SUCCEEDED: Create namespace external in 14.738922ms === 2025-08-04T18:09:41.290578Z info tf === BEGIN: Deploy echo instances === 2025-08-04T18:09:41.342585Z error tf === FAILED: Deploy echo instances === 2025-08-04T18:09:41.342606Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82/_suite_context/Service-ServiceAccount.3525837110.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:09:41.342652Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:41.342686Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:41.342708Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:41.412433Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:41.461170Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:41.520409Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:41.537848Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:41.593566Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:27Z/554d0b1448-3c6d-4af1-9ae3-f6da61cd4f86 2025-08-04T18:09:41.594222Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2511s 2025-08-04T18:09:41.594676Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:41.744747Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82/_suite_context/Service-ServiceAccount.3525837110.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:09:41.744811Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82/_suite_context/Service-ServiceAccount.3525837110.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T18:09:41.744819Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8cb3b82/_suite_context/Service-ServiceAccount.3525837110.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T18:09:41.744853Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:41.744852Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:41.744886Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:09:41.787496Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:41.802496Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:41.813536Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:09:27Z/55f62a7e6e-6461-47d3-a2a6-9b5970e5e560 2025-08-04T18:09:41.813596Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4706s 2025-08-04T18:09:41.814058Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:41.986680Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:42.019389Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:42.049608Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T18:09:42.049673Z info tf === SUCCEEDED: Cleanup Istio in 58.41µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root DONE 2 runs, 25 tests, 12 failures in 303.409s exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$/^naked\[foo\]->server:http-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 3056173 2025-08-04T18:09:55.141640Z warn unable to resolve TARGET_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:09:55.141692Z warn unable to resolve LOCAL_OUT. Dir /root/mohit/istio/out/linux_ppc64le does not exist 2025-08-04T18:09:55.172812Z info tf === Test Framework Settings === 2025-08-04T18:09:55.172933Z info tf TestID: security_ca_custom_root RunID: 95aa94fe-6c47-4953-be2a-f2740e0c80db NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T18:09:55.172939Z info tf =============================== 2025-08-04T18:09:55.173123Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-95aa94f 2025-08-04T18:09:55.173212Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T18:09:55.173225Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T18:09:55.173242Z info tf Using KubeConfigs: [/root/upi/cA/auth/kubeconfig]. 2025-08-04T18:09:55.173250Z info tf === BEGIN: Building clusters === 2025-08-04T18:09:55.179341Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cA/auth/kubeconfig 2025-08-04T18:09:55.179422Z info tf === DONE: Building clusters === 2025-08-04T18:09:55.179443Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T18:09:55.286669Z info configmap istio-ca-root-cert is deleted 2025-08-04T18:09:55.286728Z info tf === SUCCESS: context setup function === 2025-08-04T18:09:55.286736Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T18:09:55.286755Z info tf === Istio Component Config === 2025-08-04T18:09:55.286817Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T18:09:55.286823Z info tf ================================ 2025-08-04T18:09:55.289923Z info tf skipping deployment as specified in the config 2025-08-04T18:09:55.289942Z info tf === SUCCEEDED: Deploy Istio in 3.199551ms [Suite=security_ca_custom_root]=== 2025-08-04T18:09:55.289952Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T18:09:55.316730Z info tf === SUCCEEDED: Create namespace echo1 in 26.764707ms === Generating RSA private key, 2048 bit long modulus (2 primes) ..........................+++++ ............+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-95aa94f/_suite_context/ca-custom-root-886713607/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ...........+++++ .....................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-95aa94f/_suite_context/ca-custom-root-886713607/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) .......+++++ ....................+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-95aa94f/_suite_context/ca-custom-root-886713607/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T18:09:55.579353Z info tf === BEGIN: Create namespace external === 2025-08-04T18:09:55.650222Z info tf === SUCCEEDED: Create namespace external in 70.863338ms === 2025-08-04T18:09:55.860715Z info tf === BEGIN: Deploy echo instances === 2025-08-04T18:09:57.127203Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.141966Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.142381Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.142583Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.160515Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.197612Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.216765Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.220468Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.224842Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.227705Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.231175Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.242985Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.243086Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.243083Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.261718Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.265707Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.267386Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.268962Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.275729Z info klog Waiting for caches to sync controller=echo 2025-08-04T18:09:57.298439Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.317703Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.321351Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.325919Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.331779Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.365982Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.368582Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.369983Z info klog Caches are synced controller=echo 2025-08-04T18:09:57.375916Z info klog Caches are synced controller=echo 2025-08-04T18:10:10.082206Z info tf === SUCCEEDED: Deploy echo instances in 14.221479723s === 2025-08-04T18:10:10.150918Z info tf === DONE: Setup: 'security_ca_custom_root' (14.971462119s) === 2025-08-04T18:10:10.150965Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-08-04T18:10:10.199721Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T18:10:40.200429Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000700989s)' === 2025-08-04T18:10:40.200507Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:10:40.200513Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:10:40.200580Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:10:40.224678Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:10:40.260118Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:10:40.391076Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:40.414803Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:40.575395Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72103b8c23-8448-4496-a934-071bbe6af9a6 2025-08-04T18:10:40.575743Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3749s 2025-08-04T18:10:40.576248Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:40.685329Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/729bfa37f9-c204-4e52-9f96-7bb32c91f68e 2025-08-04T18:10:40.685508Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4847s 2025-08-04T18:10:40.686064Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:40.798286Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/7266d646d9-17c2-4dc1-b74f-80176d3344de 2025-08-04T18:10:40.798362Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5975s 2025-08-04T18:10:40.798814Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (30.86s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 2) (30.86s) === RUN TestTrustDomainValidation/From_primary-0 2025-08-04T18:10:10.198991Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T18:10:41.057817Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (30.858818228s)' === 2025-08-04T18:10:41.057943Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:10:41.057954Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:10:41.057954Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:10:41.078998Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:10:41.098772Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:10:41.265788Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:41.281691Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:41.318548Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72b24f6641-423b-4859-9714-aff8ad967de1 2025-08-04T18:10:41.318846Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2606s 2025-08-04T18:10:41.319383Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:41.429422Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/721e420857-5110-48da-93f9-c22d3d90be7b 2025-08-04T18:10:41.429549Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3713s 2025-08-04T18:10:41.430341Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:41.551623Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72f63993a8-7878-431d-b388-21112562a07f 2025-08-04T18:10:41.551788Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4935s 2025-08-04T18:10:41.552294Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (31.69s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (re-run 2) (31.69s) === RUN TestTrustDomainValidation 2025-08-04T18:10:10.151388Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T18:10:41.885182Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (31.733786034s)' === 2025-08-04T18:10:41.885281Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T18:10:41.885312Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T18:10:41.885301Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T18:10:41.945117Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:10:42.054610Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:10:42.244531Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:42.293448Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:42.343619Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/726ae240ac-f5f9-4548-b5ab-ab3691a2ba1a 2025-08-04T18:10:42.343901Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4584s 2025-08-04T18:10:42.344404Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:42.620990Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/7249541103-b6c9-4316-8b62-f9428ee07e84 2025-08-04T18:10:42.621247Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.7357s 2025-08-04T18:10:42.621919Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:42.872987Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72b012bc5f-87d0-4076-bfeb-d02c0fd98681 2025-08-04T18:10:42.873141Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9876s 2025-08-04T18:10:42.873838Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation (34.07s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (re-run 2) (34.07s) 2025-08-04T18:10:44.218923Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-08-04T18:10:44.218976Z info tf === Suite "security_ca_custom_root" run time: 49.039537979s === 2025-08-04T18:10:44.218989Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-08-04T18:10:44.219357Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T18:10:44.219384Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T18:10:44.219375Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T18:10:44.366892Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:10:44.457762Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:10:44.587943Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:44.622075Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:44.685783Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:44Z/73fa242080-0e70-441d-bc4a-d14390af651d 2025-08-04T18:10:44.685945Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4662s 2025-08-04T18:10:44.686532Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:44.815993Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:44Z/7383ca4475-34cc-4345-b9d1-4995d7779b4a 2025-08-04T18:10:44.816067Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5963s 2025-08-04T18:10:44.816558Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:44.940550Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:44Z/73b054116e-a40b-468a-9de2-a9b67ebba50c 2025-08-04T18:10:44.940699Z info tf istioctl ([x internal-debug --all configz]): completed after 0.7209s 2025-08-04T18:10:44.941141Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:45.327005Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T18:10:45.327129Z info tf === SUCCEEDED: Cleanup Istio in 112.998µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root === Failed === FAIL: tests/integration/security/ca_custom_root TestSecureNaming (11.08s) 2025-08-04T18:04:59.826790Z info tf === BEGIN: Test: 'security_ca_custom_root[TestSecureNaming]' === secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:114: retry.UntilSuccessOrFail: timeout while waiting after 10 attempts (last error: CA certificate (-----BEGIN CERTIFICATE----- MIIC/DCCAeSgAwIBAgIQTUjJ+mZ0yr7AX2hpmhvyijANBgkqhkiG9w0BAQsFADAY MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTI1MDgwNDE4MDMyNFoXDTM1MDgw MjE4MDMyNFowGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBAN3CS/Jf1TezHvDFoIQ6+C4//pRvjHH3o3Uh3GVJ fg2YtrHmj4iq9Eq9zloergCXnnNcqFagVdynEuqurTMFF1gD8kbKq+8ozYaD18M/ 9f++FR8WBstpC4HT1TSd/YSGrjCfuiU37xdJEeJG6QkzQhz4J+7bVR8o/6CSVH7Z NDsqrolQ09+RyvBlDFUemVm+OZ7ZeC3je7b/pKl5Uscf6mbKR52iQdAYXACKONrk mj3xk7d0j66nzXRjWOSSS5Is5ULRH22P2cqkyk6QNdxHxg4sn0oyVVhM9q4QWXQH WlWElJxnMWRTkkVk4OTg0XKFLM1ZKKgZFWQ2zkypGBUbjNkCAwEAAaNCMEAwDgYD VR0PAQH/BAQDAgIEMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNnL/E0WEoZP XFVnyiqtL1HuWxyAMA0GCSqGSIb3DQEBCwUAA4IBAQCkPML6p6LFIIddOUsRAlCG GcZPPkpbViSer27UsOG/4HEWzp3irPfLXQZI7Aoa26NYYqwnLHya5WSrER6joffa 7jiApP9hFND3PLOQdoe74VCVep+Q2orjmv3kKRojvYn3KrJp8vXzdk71tjYMUXo0 x6V2toTPupClFajehMoOA6PBQ/0jjHMdj5szXndBlPD0g9duWx4I7pY9x2ZOCr3u 5KOmx4CephIaURg6F70HMSrKlLHyEc8zKg1mNmNDCKktwJ6ki495+6tUQ9LaQ+Wh iZBSq2nKF55D4hTvMbIxcALMb3Zwjw4HAvBmQm28wuxLhf4VljgZhZQ0UOnWntfS -----END CERTIFICATE----- ) not matching plugin cert (-----BEGIN CERTIFICATE----- MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN 8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a 3g== -----END CERTIFICATE----- )) 2025-08-04T18:05:09.835104Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestSecureNaming] (10.008306871s)' === 2025-08-04T18:05:09.835226Z error tf === Dumping Namespace external State for TestSecureNaming... 2025-08-04T18:05:09.835240Z error tf === Dumping Namespace echo1 State for TestSecureNaming... 2025-08-04T18:05:09.835226Z error tf === Dumping Istio Deployment State for TestSecureNaming... 2025-08-04T18:05:09.881699Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:09.945647Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:10.060916Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:10.111689Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:10.175100Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:04:59Z/16a0f063ea-af71-4582-9599-4fd03cae85f7 2025-08-04T18:05:10.175268Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3396s 2025-08-04T18:05:10.175886Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:10.330102Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:04:59Z/16edcbbdb9-3e20-494e-8f93-ff1806c0f41b 2025-08-04T18:05:10.330344Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4947s 2025-08-04T18:05:10.330746Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:10.460325Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:04:59Z/16caccc4ff-99af-4e35-a73a-9996349845c5 2025-08-04T18:05:10.460447Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6248s 2025-08-04T18:05:10.460861Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:10.796680Z error tf === Dumping Namespace istio-system State for TestSecureNaming... 2025-08-04T18:05:10.807975Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:10.816942Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:10.886987Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:10.889164Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (30.97s) 2025-08-04T18:05:10.933861Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.23.1.237:39386->172.30.189.239:443: read: connection reset by peer 2025-08-04T18:05:40.934361Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.000495869s)' === 2025-08-04T18:05:40.934461Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T18:05:40.934478Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T18:05:40.934470Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T18:05:41.000386Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:41.059199Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:41.236540Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:41.236554Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:41.310654Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17e282a81d-b220-4714-adb8-91a2429b5659 2025-08-04T18:05:41.310876Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3759s 2025-08-04T18:05:41.311498Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:41.476479Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/178852609f-bc5e-4635-9a28-58495e7aea03 2025-08-04T18:05:41.476600Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5416s 2025-08-04T18:05:41.477220Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:41.621924Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/173c290d34-fe09-43fe-9a55-e9707af456e6 2025-08-04T18:05:41.622026Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6870s 2025-08-04T18:05:41.622830Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0 (31.98s) 2025-08-04T18:05:10.933594Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T18:05:41.932888Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (30.999294289s)' === 2025-08-04T18:05:41.932936Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T18:05:41.932963Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T18:05:41.932963Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T18:05:41.960638Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:42.012476Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:42.155758Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:42.155885Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:42.260325Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17aad80cc9-45a2-4654-b398-8fe917b54873 2025-08-04T18:05:42.260590Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3273s 2025-08-04T18:05:42.262219Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:42.417949Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/1748acd39a-0bd0-49f9-87b1-1ab8621574c5 2025-08-04T18:05:42.418052Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4847s 2025-08-04T18:05:42.418799Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:42.553866Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17e96147b8-eec4-49b7-b314-060e1ab60fec 2025-08-04T18:05:42.553989Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6206s 2025-08-04T18:05:42.554690Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming (34.53s) 2025-08-04T18:05:10.902987Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T18:05:42.918118Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.015122788s)' === 2025-08-04T18:05:42.918241Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T18:05:42.918241Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T18:05:42.918257Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T18:05:43.053491Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:05:43.185825Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:05:43.392390Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:43.433119Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:05:43.511464Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17c37e47a9-1e2e-4603-b04d-25a58e60ff87 2025-08-04T18:05:43.511790Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5934s 2025-08-04T18:05:43.512321Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:44.085091Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/17043b7737-500d-4696-ab89-f68f74134d45 2025-08-04T18:05:44.085226Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.1668s 2025-08-04T18:05:44.086014Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:05:44.525352Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:11Z/1766f657f3-ad86-4d33-a7b0-2cefb08bc451 2025-08-04T18:05:44.525519Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.6070s 2025-08-04T18:05:44.526073Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (31.50s) 2025-08-04T18:05:46.580448Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === 2025-08-04T18:05:58.783412Z error port forward failed: lost connection to pod 2025-08-04T18:05:58.814887Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward terminated 2025-08-04T18:05:58.838276Z error port forward failed: lost connection to pod 2025-08-04T18:05:58.870493Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward terminated 2025-08-04T18:05:58.879708Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward reconnect success 2025-08-04T18:05:58.941785Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward reconnect success 2025-08-04T18:05:59.155648Z error port forward failed: lost connection to pod 2025-08-04T18:05:59.230288Z error port forward failed: lost connection to pod 2025-08-04T18:05:59.369553Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward terminated 2025-08-04T18:05:59.431943Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward reconnect success 2025-08-04T18:05:59.441407Z warn pod: external/external-v1-987785fb7-78rs2 port forward terminated 2025-08-04T18:05:59.503990Z warn pod: external/external-v1-987785fb7-78rs2 port forward reconnect success 2025-08-04T18:06:01.246717Z error port forward failed: lost connection to pod 2025-08-04T18:06:01.392945Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward terminated 2025-08-04T18:06:01.490666Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward reconnect success 2025-08-04T18:06:02.057184Z error port forward failed: lost connection to pod 2025-08-04T18:06:02.174510Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward terminated 2025-08-04T18:06:02.232712Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward reconnect success 2025-08-04T18:06:02.879757Z error port forward failed: lost connection to pod 2025-08-04T18:06:02.966306Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward terminated 2025-08-04T18:06:03.051682Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward reconnect success 2025-08-04T18:06:03.059040Z error port forward failed: lost connection to pod 2025-08-04T18:06:03.088345Z error port forward failed: lost connection to pod 2025-08-04T18:06:03.230854Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward terminated 2025-08-04T18:06:03.289302Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward reconnect success 2025-08-04T18:06:03.477416Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-08-04T18:06:03.537657Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-08-04T18:06:03.874399Z error port forward failed: lost connection to pod 2025-08-04T18:06:04.029392Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward terminated 2025-08-04T18:06:04.126109Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward reconnect success 2025-08-04T18:06:04.163826Z error port forward failed: lost connection to pod 2025-08-04T18:06:04.164084Z error port forward failed: lost connection to pod 2025-08-04T18:06:04.371101Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward terminated 2025-08-04T18:06:04.425896Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward terminated 2025-08-04T18:06:04.431931Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward reconnect success 2025-08-04T18:06:04.483187Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T18:06:16.581577Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.001122914s)' === 2025-08-04T18:06:16.581665Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:06:16.581686Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:06:16.581688Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:06:16.637220Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:06:16.681500Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:06:16.834526Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:16.834555Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:16.912600Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20bc87a754-4076-47c0-8467-dec9b7fbc624 2025-08-04T18:06:16.912910Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3309s 2025-08-04T18:06:16.913520Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:17.184890Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20a190fd16-1d39-4ca7-81b8-bcbe9ae7fa6e 2025-08-04T18:06:17.184978Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6029s 2025-08-04T18:06:17.185532Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:17.570111Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20cf615e1b-62d4-43d1-8b26-9838155ccb57 2025-08-04T18:06:17.570245Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9882s 2025-08-04T18:06:17.570919Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (31.29s) 2025-08-04T18:06:18.076857Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T18:06:48.078165Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.00129715s)' === 2025-08-04T18:06:48.078268Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T18:06:48.078306Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T18:06:48.078309Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T18:06:48.156639Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: image can't be pulled 2025-08-04T18:06:48.208069Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:06:48.364678Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:48.390363Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:06:48.406085Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20fe7fd8e8-4a47-4dc0-b02d-1d565cf90127 2025-08-04T18:06:48.406361Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3277s 2025-08-04T18:06:48.407148Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:48.549294Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20eb085a4e-9844-433d-89b2-59176c183d22 2025-08-04T18:06:48.549448Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4707s 2025-08-04T18:06:48.550087Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:06:48.830870Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20c288650c-a2b6-42fe-99b1-cb0cd112f0bd 2025-08-04T18:06:48.831021Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.7522s 2025-08-04T18:06:48.831630Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.90s) 2025-08-04T18:06:49.367643Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === 2025-08-04T18:06:51.832983Z error port forward failed: lost connection to pod 2025-08-04T18:06:52.033322Z warn pod: echo1/client-v1-9cfd95bd4-dwxrh port forward terminated 2025-08-04T18:06:52.621434Z warn pod: echo1/client-v1-9cfd95bd4-dwxrh port forward reconnect success 2025-08-04T18:07:04.883745Z error port forward failed: lost connection to pod 2025-08-04T18:07:04.946154Z error port forward failed: lost connection to pod 2025-08-04T18:07:05.315277Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward terminated 2025-08-04T18:07:05.370441Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward terminated 2025-08-04T18:07:05.380078Z warn pod: echo1/server-naked-foo-v1-6f468cf89c-45gr6 port forward reconnect success 2025-08-04T18:07:05.431566Z warn pod: echo1/server-naked-bar-v1-7cf5cd474f-dt77l port forward reconnect success 2025-08-04T18:07:05.439004Z error port forward failed: lost connection to pod 2025-08-04T18:07:05.508086Z error port forward failed: lost connection to pod 2025-08-04T18:07:05.870593Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward terminated 2025-08-04T18:07:05.941279Z warn pod: external/external-v1-987785fb7-78rs2 port forward terminated 2025-08-04T18:07:05.958086Z warn pod: echo1/server-naked-foo-alt-v1-6cbc84d88f-rljcz port forward reconnect success 2025-08-04T18:07:06.016343Z warn pod: external/external-v1-987785fb7-78rs2 port forward reconnect success 2025-08-04T18:07:07.496012Z error port forward failed: lost connection to pod 2025-08-04T18:07:07.893541Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward terminated 2025-08-04T18:07:07.954786Z warn pod: echo1/proxyless-grpc-v1-759745c5b9-m64dp port forward reconnect success 2025-08-04T18:07:08.237374Z error port forward failed: lost connection to pod 2025-08-04T18:07:08.673931Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward terminated 2025-08-04T18:07:08.732446Z warn pod: echo1/sotw-v1-c494d6b8d-mljmj port forward reconnect success 2025-08-04T18:07:09.057065Z error port forward failed: lost connection to pod 2025-08-04T18:07:09.294235Z error port forward failed: lost connection to pod 2025-08-04T18:07:09.465408Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward terminated 2025-08-04T18:07:09.541543Z error port forward failed: lost connection to pod 2025-08-04T18:07:09.544625Z warn pod: echo1/b-v1-5bccd7df8d-lnwqj port forward reconnect success 2025-08-04T18:07:09.730312Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward terminated 2025-08-04T18:07:09.799023Z warn pod: echo1/server-v1-6d77f6b7c4-2v679 port forward reconnect success 2025-08-04T18:07:09.978020Z warn pod: echo1/statefulset-v1-0 port forward terminated 2025-08-04T18:07:10.044694Z warn pod: echo1/statefulset-v1-0 port forward reconnect success 2025-08-04T18:07:10.131438Z error port forward failed: lost connection to pod 2025-08-04T18:07:10.437279Z error port forward failed: lost connection to pod 2025-08-04T18:07:10.489797Z error port forward failed: lost connection to pod 2025-08-04T18:07:10.529056Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward terminated 2025-08-04T18:07:10.589129Z warn pod: echo1/c-v1-5f56c57bb9-pgbr7 port forward reconnect success 2025-08-04T18:07:10.871186Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward terminated 2025-08-04T18:07:10.926460Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward terminated 2025-08-04T18:07:10.954609Z warn pod: echo1/headless-v1-5455bcc7b9-wbrlx port forward reconnect success 2025-08-04T18:07:11.002504Z warn pod: echo1/a-v1-6b586f9d47-mjz5w port forward reconnect success test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.3.203:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.3.203:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T18:07:19.368367Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000718998s)' === 2025-08-04T18:07:19.368468Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T18:07:19.368467Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T18:07:19.368488Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T18:07:19.838480Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:19.903970Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:20.193162Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:20.195028Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:20.230860Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/2033f67e26-dd71-4563-844f-17e4ee7b0844 2025-08-04T18:07:20.231104Z info tf istioctl ([x internal-debug --all configz]): completed after 0.8621s 2025-08-04T18:07:20.231781Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:20.633490Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/201ba1c2dd-1592-4b1c-b9bd-7ddfb457cc7c 2025-08-04T18:07:20.633747Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.2647s 2025-08-04T18:07:20.634338Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:20.767405Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20242d8594-874e-410b-b144-7b1f5de95f0d 2025-08-04T18:07:20.767549Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.3986s 2025-08-04T18:07:20.768338Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (96.60s) 2025-08-04T18:05:45.648659Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T18:07:21.265321Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m35.616652377s)' === 2025-08-04T18:07:21.265443Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:07:21.265455Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:07:21.265478Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:07:21.305371Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:21.394947Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:21.460320Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:21.491028Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:21.547229Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20096c9bff-56fc-43a3-9b6f-d6fe4f2c531c 2025-08-04T18:07:21.547374Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2817s 2025-08-04T18:07:21.548157Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:21.682008Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/2027b05975-8b65-43a3-a6b9-57fbb8cb8c78 2025-08-04T18:07:21.682165Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4165s 2025-08-04T18:07:21.682816Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:21.847583Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/202bdf121a-1378-4f79-9ae0-1fda0efe9aa7 2025-08-04T18:07:21.847835Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5822s 2025-08-04T18:07:21.848519Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (97.84s) 2025-08-04T18:05:45.434860Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T18:07:22.246134Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m36.811259564s)' === 2025-08-04T18:07:22.246277Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T18:07:22.246276Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T18:07:22.246300Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T18:07:22.305501Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:07:22.335267Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:07:22.462860Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:22.472184Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:07:22.523950Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/208a325929-27ba-442f-95b8-a1390a8004ee 2025-08-04T18:07:22.524193Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2777s 2025-08-04T18:07:22.524642Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:22.660667Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/205ad94894-cc06-419d-9061-2eb2000b8c38 2025-08-04T18:07:22.660797Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4142s 2025-08-04T18:07:22.661382Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:07:22.823614Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:05:45Z/20603c513a-8686-4df2-b250-fbcec06377c3 2025-08-04T18:07:22.823831Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5772s 2025-08-04T18:07:22.824481Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 1) (32.24s) 2025-08-04T18:08:31.747100Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T18:09:01.747576Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000472033s)' === 2025-08-04T18:09:01.747669Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:09:01.747686Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:09:01.747696Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:09:01.987644Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:02.033547Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:02.328922Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:02.915766Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:03.008708Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4991db33cc-f800-4fc5-9baf-af2405bd0cc2 2025-08-04T18:09:03.008971Z info tf istioctl ([x internal-debug --all configz]): completed after 1.2609s 2025-08-04T18:09:03.009567Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:03.477437Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/49692d828b-790e-4cc9-8d22-554f92780d5d 2025-08-04T18:09:03.477611Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.7295s 2025-08-04T18:09:03.478247Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:03.603458Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/49a4c02fd2-c3cb-4dc0-a0e1-ae1ac9e0149f 2025-08-04T18:09:03.603540Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.8554s 2025-08-04T18:09:03.604119Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (re-run 1) (33.43s) 2025-08-04T18:08:31.746415Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T18:09:03.985511Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (32.239089047s)' === 2025-08-04T18:09:03.985621Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:09:03.985644Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:09:03.985676Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:09:04.083746Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:04.188910Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:04.520822Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:04.578336Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:04.608329Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4938668aa3-1893-4440-8607-27e9b74b09a0 2025-08-04T18:09:04.608539Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5892s 2025-08-04T18:09:04.609182Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:04.851608Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4988df792d-151e-48c7-80ff-a46089ba9c7a 2025-08-04T18:09:04.851779Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8324s 2025-08-04T18:09:04.852308Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:04.972279Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/4916e9157e-f961-4d28-a140-ba31e00941c3 2025-08-04T18:09:04.972446Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9530s 2025-08-04T18:09:04.972896Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (re-run 1) (37.32s) 2025-08-04T18:08:31.716537Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T18:09:05.181129Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (33.464577859s)' === 2025-08-04T18:09:05.181261Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T18:09:05.181261Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T18:09:05.181290Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T18:09:05.203160Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:09:05.294066Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:09:05.418135Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:05.467479Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:09:05.856196Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/494705aba9-d35a-45e5-b638-67cd9fe08e98 2025-08-04T18:09:05.856447Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6749s 2025-08-04T18:09:05.857117Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:07.137077Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/493cac5504-bfe7-40c7-9c21-bec8706eedfc 2025-08-04T18:09:07.137211Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.9557s 2025-08-04T18:09:07.137864Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:09:07.844998Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:08:31Z/49577646a9-8a6e-4097-80c9-89a47a0d7ae4 2025-08-04T18:09:07.845124Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.6636s 2025-08-04T18:09:07.845675Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 2) (30.86s) 2025-08-04T18:10:10.199721Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T18:10:40.200429Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000700989s)' === 2025-08-04T18:10:40.200507Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:10:40.200513Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:10:40.200580Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T18:10:40.224678Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:10:40.260118Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:10:40.391076Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:40.414803Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:40.575395Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72103b8c23-8448-4496-a934-071bbe6af9a6 2025-08-04T18:10:40.575743Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3749s 2025-08-04T18:10:40.576248Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:40.685329Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/729bfa37f9-c204-4e52-9f96-7bb32c91f68e 2025-08-04T18:10:40.685508Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4847s 2025-08-04T18:10:40.686064Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:40.798286Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/7266d646d9-17c2-4dc1-b74f-80176d3344de 2025-08-04T18:10:40.798362Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5975s 2025-08-04T18:10:40.798814Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (re-run 2) (31.69s) 2025-08-04T18:10:10.198991Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T18:10:41.057817Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (30.858818228s)' === 2025-08-04T18:10:41.057943Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:10:41.057954Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:10:41.057954Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T18:10:41.078998Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:10:41.098772Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:10:41.265788Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:41.281691Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:41.318548Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72b24f6641-423b-4859-9714-aff8ad967de1 2025-08-04T18:10:41.318846Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2606s 2025-08-04T18:10:41.319383Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:41.429422Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/721e420857-5110-48da-93f9-c22d3d90be7b 2025-08-04T18:10:41.429549Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3713s 2025-08-04T18:10:41.430341Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:41.551623Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72f63993a8-7878-431d-b388-21112562a07f 2025-08-04T18:10:41.551788Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4935s 2025-08-04T18:10:41.552294Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (re-run 2) (34.07s) 2025-08-04T18:10:10.151388Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T18:10:41.885182Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (31.733786034s)' === 2025-08-04T18:10:41.885281Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T18:10:41.885312Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T18:10:41.885301Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T18:10:41.945117Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-54f8f7b6c5-nnjck/istio-proxy for: container "istio-proxy" in pod "istio-egressgateway-54f8f7b6c5-nnjck" is waiting to start: trying and failing to pull image 2025-08-04T18:10:42.054610Z warn tf Unable to get logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-b7b89b8d5-6g9jk/istio-proxy for: container "istio-proxy" in pod "istio-ingressgateway-b7b89b8d5-6g9jk" is waiting to start: trying and failing to pull image 2025-08-04T18:10:42.244531Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:42.293448Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Pending) 2025-08-04T18:10:42.343619Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/726ae240ac-f5f9-4548-b5ab-ab3691a2ba1a 2025-08-04T18:10:42.343901Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4584s 2025-08-04T18:10:42.344404Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:42.620990Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/7249541103-b6c9-4316-8b62-f9428ee07e84 2025-08-04T18:10:42.621247Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.7357s 2025-08-04T18:10:42.621919Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T18:10:42.872987Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T18:10:10Z/72b012bc5f-87d0-4076-bfeb-d02c0fd98681 2025-08-04T18:10:42.873141Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.9876s 2025-08-04T18:10:42.873838Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF DONE 3 runs, 28 tests, 15 failures in 366.687s exec: go version