exec: [go test -json -tags=integ -timeout 180m ./tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 468599 2025-08-04T14:55:17.860214Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:55:17.860273Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:55:17.890601Z info tf === Test Framework Settings === 2025-08-04T14:55:17.890731Z info tf TestID: security_ca_custom_root RunID: 027f5d2e-c049-41ea-95e8-2fd15780a5fa NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T14:55:17.890738Z info tf =============================== 2025-08-04T14:55:17.890892Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2 2025-08-04T14:55:17.890987Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T14:55:17.891000Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T14:55:17.891005Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T14:55:17.891021Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T14:55:17.891030Z info tf === BEGIN: Building clusters === 2025-08-04T14:55:17.892684Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T14:55:17.892692Z info tf === DONE: Building clusters === 2025-08-04T14:55:17.892710Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T14:55:17.937882Z info configmap istio-ca-root-cert is deleted 2025-08-04T14:55:17.937969Z info tf === SUCCESS: context setup function === 2025-08-04T14:55:17.937979Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T14:55:17.937995Z info tf === Istio Component Config === 2025-08-04T14:55:17.938040Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T14:55:17.938046Z info tf ================================ 2025-08-04T14:55:17.941966Z info tf skipping deployment as specified in the config 2025-08-04T14:55:17.941993Z info tf === SUCCEEDED: Deploy Istio in 4.00571ms [Suite=security_ca_custom_root]=== 2025-08-04T14:55:17.942003Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T14:55:18.021496Z info tf === SUCCEEDED: Create namespace echo1 in 79.486407ms === Generating RSA private key, 2048 bit long modulus (2 primes) ***************************************************************+++++ *************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/_suite_context/ca-custom-root-677743191/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***********+++++ ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/_suite_context/ca-custom-root-677743191/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************+++++ **********************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/_suite_context/ca-custom-root-677743191/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T14:55:19.299766Z info tf === BEGIN: Create namespace external === 2025-08-04T14:55:19.317894Z info tf === SUCCEEDED: Create namespace external in 18.16821ms === 2025-08-04T14:55:19.397105Z info tf === BEGIN: Deploy echo instances === 2025-08-04T14:55:19.743469Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.751955Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.752027Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.765263Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.778790Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.784855Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.785720Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.802415Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.806878Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.807691Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.844520Z info klog Caches are synced controller=echo 2025-08-04T14:55:19.853193Z info klog Caches are synced controller=echo 2025-08-04T14:55:19.853266Z info klog Caches are synced controller=echo 2025-08-04T14:55:19.865831Z info klog Caches are synced controller=echo 2025-08-04T14:55:19.879950Z info klog Caches are synced controller=echo 2025-08-04T14:55:19.885539Z info klog Caches are synced controller=echo 2025-08-04T14:55:19.885837Z info klog Caches are synced controller=echo 2025-08-04T14:55:19.908821Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.911678Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:19.911963Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:20.002600Z info klog Caches are synced controller=echo 2025-08-04T14:55:20.007226Z info klog Caches are synced controller=echo 2025-08-04T14:55:20.008625Z info klog Caches are synced controller=echo 2025-08-04T14:55:20.570512Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:55:20.609765Z info klog Caches are synced controller=echo 2025-08-04T14:55:21.370852Z info klog Caches are synced controller=echo 2025-08-04T14:55:21.412527Z info klog Caches are synced controller=echo 2025-08-04T14:55:21.412523Z info klog Caches are synced controller=echo 2025-08-04T14:55:43.917278Z info tf === SUCCEEDED: Deploy echo instances in 24.520154791s === 2025-08-04T14:55:43.950123Z info tf === DONE: Setup: 'security_ca_custom_root' (26.057394415s) === 2025-08-04T14:55:43.950196Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === PASS tests/integration/security/ca_custom_root.TestMultiRootSetup/From_primary-0/server:server-naked-foo-alt[success] (1.06s) PASS tests/integration/security/ca_custom_root.TestMultiRootSetup/From_primary-0 (1.06s) PASS tests/integration/security/ca_custom_root.TestMultiRootSetup (1.09s) === RUN TestSecureNaming 2025-08-04T14:55:45.036577Z info tf === BEGIN: Test: 'security_ca_custom_root[TestSecureNaming]' === secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:114: retry.UntilSuccessOrFail: timeout while waiting after 10 attempts (last error: CA certificate (-----BEGIN CERTIFICATE----- MIIFFDCCAvygAwIBAgIUbQ/RQ9GNnLx7rNVNUJHeizdO1TswDQYJKoZIhvcNAQEL BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjMxMjEx MTYyNDM2WhcNMzMxMjA4MTYyNDM2WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAN7V2l7b /bZsHlWY5qC1p99Au3I0VYr34i/2BIMHSnh1jHn1B6qspjHw2IkqH0SU41M3Vpr7 m06HGuuZHsq8biUfuUL6Ywt+Vyj2UnPuc6SDPRXxAkBaVGGEAs30akJPkhbbA2qv jFxfc2A8GCfnHpGYQkZOvZEQNeHPYkTDsdLEKdbdXkzJMFEGxGDgYiW7xjowDkta YlrLBIeHwRCnZ/gTjdCu6oPjCW9JsySbbfx8dYWthIDf/SGqnX8gnDn5K1W5Ip+w xaiXZ9FsEzrWd+DXx67sPAS2wl/Qe+RVFLbCGfGnfKdj0CGUtwV/hE2BLWk7fMK/ dzQR9Bc6GM06uJejZFGuVrwkJldzWW7rp6cJc5D5vJxYCRZZsuk3x6o/mg+z+meh kYeUCgsEGU1eKJB7tJ6myiYCLvgjihnnbS0fQlq+9XhHh3IlQfYvZsgPPlWZjyE6 SO0JUNgCenuvgawihEHCUvFW/VbeuhxNlcP8w03tXcZCPJd75Xe1GRboa0KEcCwx hFMWMZmqLQOZ6XZUAWZIPqKPvBCLJKCDhXwf3kZjX9Geg04+KJwsJt5wJndlWjdb LWUIoiy41Jovc3N2hME5cinTllZI4kdOrnZ+NdL5aqoppfjZDQGHESuuPY0vxJEP dcfM5Q3BiMo/K/hNCnbp1ywtvbzGqd8/b9OtAgMBAAGjQjBAMB0GA1UdDgQWBBS9 7L7n9MlfjNgR6RLw5qN+49OOYDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEANnOOVzb6CpOkkXsSwxqaFfrCwJT+L+1O H8WvVtewAmzu8TcfuVjwwSgnVMF48TbnA1KvHIWktRNGtSTyEMLN5pH4SvCyK/9w IeIOqTNyHmN/4nETT7HjbszdjfsIWYZAjcq2cyhRj+650OJdotUHbBo7iBbAi4TW IuqseH4hgrT7qDFebhcVz8ualx9UJkmR7QqWhWqOsMbcLtm4OLZCFNJseaHM/t6H seO/OH7FNqe+GMCoXX1m348iJBZ/1ZhBMReqtNMQKpGL23fQqJnUTrD3717kS+xF 36PsFP5YYtBJ9WMXfX8Wk/YQIimXpcQeuF40VAIw5EG6SxntpBjoSPET2mTj2j9X Gunoh3PjYMBOlszrQVs66Qwe+vBj6xdPMHyRI0eTiETnmVG5jUo2cJnr8ztqD6oU zCmTX2SXCE9FXJ6MifPHEXJ1OPb9IY5tcUdjxhIB2HAigFHasxBlX44Ob6nHUyJO ZqhsF8FouuzSQTRWP7soIA3HT9MlZHhRh3Uef+ZXziKZR/Uyb66uYylIWCWsAoiU Hw4wPZhG65XTVQfg7R2cz1w72v7ugXAy3xbGfgLXhFzVnjy/ttIV/TSgpdBquhTQ 1ZG9Ieiu16avJkphqkmNr/CRCdZqDGp4DGQZNo69POGphapVe7EZenpzDU787oZF tugW0JVzMZI= -----END CERTIFICATE----- ) not matching plugin cert (-----BEGIN CERTIFICATE----- MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN 8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a 3g== -----END CERTIFICATE----- )) 2025-08-04T14:55:55.043584Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestSecureNaming] (10.007001878s)' === 2025-08-04T14:55:55.043692Z error tf === Dumping Namespace external State for TestSecureNaming... 2025-08-04T14:55:55.043706Z error tf === Dumping Istio Deployment State for TestSecureNaming... 2025-08-04T14:55:55.043710Z error tf === Dumping Namespace echo1 State for TestSecureNaming... 2025-08-04T14:55:55.105596Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-state2349979250/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:55:55.111272Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:55:55.134752Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-state2349979250/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:55:55.140679Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:55:55.569888Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:55:56.850226Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:45Z/6955dcc2ba-8df9-4554-ac66-a25fecbbc93e 2025-08-04T14:55:56.850701Z info tf istioctl ([x internal-debug --all configz]): completed after 1.8067s 2025-08-04T14:55:56.851231Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:55:56.982465Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:45Z/6939fc6ddd-c95d-4204-9b25-901a73de36fb 2025-08-04T14:55:56.982636Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.9386s 2025-08-04T14:55:56.983215Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:55:57.077471Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:45Z/69f6f6080c-ae9f-4b3e-b2fa-9dc3b5339291 2025-08-04T14:55:57.077601Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.0335s 2025-08-04T14:55:57.078417Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:55:57.217242Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:55:57.531771Z error tf === Dumping Namespace istio-system State for TestSecureNaming... 2025-08-04T14:55:57.552950Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-system-state1424418493/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:55:57.559061Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:55:57.570416Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-system-state1424418493/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:55:57.576526Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:55:57.633784Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:55:57.810062Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:55:57.894604Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:55:58.094703Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestSecureNaming (13.07s) FAIL tests/integration/security/ca_custom_root.TestSecureNaming (13.07s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] 2025-08-04T14:55:58.125325Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.0.175:41426->172.30.125.141:443: read: connection reset by peer 2025-08-04T14:56:28.126174Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.000847115s)' === 2025-08-04T14:56:28.126281Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:56:28.126282Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:56:28.126301Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:56:28.185047Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state1463783705/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:56:28.201782Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:56:28.234645Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state1463783705/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:56:28.241090Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:56:28.643055Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/7080848670-49fc-4c0e-aee4-5bfa3c21b619 2025-08-04T14:56:28.643217Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5166s 2025-08-04T14:56:28.644037Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:28.719267Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:56:28.758950Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:56:28.772230Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/7005aff96e-5f85-4be4-8c5b-51f49ea46549 2025-08-04T14:56:28.774075Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6474s 2025-08-04T14:56:28.774349Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:28.853773Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/705fff6157-8b40-429d-a097-93add59b8b31 2025-08-04T14:56:28.853922Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7273s 2025-08-04T14:56:28.854774Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:29.031752Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (31.03s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (31.03s) PASS tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-bar[fail] (0.03s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0 2025-08-04T14:55:58.125029Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T14:56:29.176896Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (31.051867468s)' === 2025-08-04T14:56:29.176952Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:56:29.176955Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:56:29.176982Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:56:29.211242Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1292889741/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:56:29.232081Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:56:29.565422Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1292889741/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:56:29.571094Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:56:29.636968Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:56:29.773831Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/70e8e56be2-759b-4a07-88a2-3e074eaec6bf 2025-08-04T14:56:29.773971Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5966s 2025-08-04T14:56:29.774546Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:29.910470Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/703ba519bc-11c5-4a92-8966-acb911c91377 2025-08-04T14:56:29.910941Z info tf istioctl ([x internal-debug --all configz]): completed after 0.7336s 2025-08-04T14:56:29.911455Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:30.033822Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/706f5a109a-3d2a-4630-9b12-fddc234a6222 2025-08-04T14:56:30.034011Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.8566s 2025-08-04T14:56:30.034644Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:30.043526Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:56:30.095499Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0 (32.45s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0 (32.45s) === RUN TestTrustDomainAliasSecureNaming 2025-08-04T14:55:58.105165Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T14:56:30.571622Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.466443941s)' === 2025-08-04T14:56:30.571755Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:56:30.571763Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:56:30.571779Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:56:30.621692Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/_test_context/istio-state3280518401/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:56:30.641316Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:56:30.671671Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/_test_context/istio-state3280518401/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:56:30.680399Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:56:30.866005Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/70796e6a40-27d7-45cb-aa2c-44ae0c2963f1 2025-08-04T14:56:30.866167Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2942s 2025-08-04T14:56:30.866852Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:30.982141Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/7085268827-0d82-4a7b-914c-8fd2852bdf55 2025-08-04T14:56:30.982261Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4103s 2025-08-04T14:56:30.982952Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:31.147542Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/70f958139d-3abb-474b-913d-b357d007148b 2025-08-04T14:56:31.148030Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5760s 2025-08-04T14:56:31.148470Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:31.189055Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:56:31.212089Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming (33.30s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming (33.30s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:http-plaintext[allow] (0.12s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:tcp-plaintext[allow] (0.07s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:http-plaintext[allow] (1.02s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:tcp-plaintext[allow] (0.02s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:http-mtls[allow] (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:tcp-mtls[allow] (0.05s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:http-mtls[deny] (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls[deny] (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls-pass-through[deny] (0.03s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-08-04T14:56:32.857563Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T14:57:02.858106Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.00053422s)' === 2025-08-04T14:57:02.858237Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T14:57:02.858237Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T14:57:02.858304Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T14:57:02.898126Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state1714524935/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:57:02.928562Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:57:02.956462Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state1714524935/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:57:02.962150Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:57:03.192825Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7173e4785f-3fb8-440a-93e1-25ac08e29063 2025-08-04T14:57:03.193024Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3343s 2025-08-04T14:57:03.194025Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:03.305823Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7173c53715-3bdd-48b2-832a-7834955d9c4c 2025-08-04T14:57:03.305954Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4473s 2025-08-04T14:57:03.307003Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:03.489969Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/711430378f-cc35-44b0-ac26-1427a63ce5da 2025-08-04T14:57:03.490530Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6318s 2025-08-04T14:57:03.490984Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:03.510725Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:57:03.533420Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (30.90s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (30.90s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] 2025-08-04T14:57:03.762236Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-08-04T14:57:03.875905Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T14:57:33.762989Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.000747991s)' === 2025-08-04T14:57:33.763083Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T14:57:33.763113Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T14:57:33.763136Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T14:57:33.817279Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3056744855/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:57:33.849211Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:57:33.874360Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3056744855/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:57:33.884270Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:57:34.037296Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71c6b50c3c-6893-4eb6-beba-b9780a110ff2 2025-08-04T14:57:34.037445Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2739s 2025-08-04T14:57:34.038271Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:34.177546Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71b46f9d2a-9fde-4705-859d-d251cdea0e61 2025-08-04T14:57:34.177885Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4143s 2025-08-04T14:57:34.178491Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:34.281082Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:57:34.379439Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71c0f497e6-d49e-4f35-b920-4c70a07d222b 2025-08-04T14:57:34.379588Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6160s 2025-08-04T14:57:34.380398Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:34.583332Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:57:34.768179Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (31.24s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (31.24s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] 2025-08-04T14:57:35.000102Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.0.178:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.0.178:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T14:58:05.000958Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000851932s)' === 2025-08-04T14:58:05.001057Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T14:58:05.001097Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T14:58:05.001081Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T14:58:05.029849Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state39573725/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:05.047180Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:05.079898Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state39573725/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:05.088538Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:05.207281Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7139c374cb-e49c-4d35-b057-56c68c687135 2025-08-04T14:58:05.207484Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2060s 2025-08-04T14:58:05.208356Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:05.374120Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/715a9616b4-9e80-4d85-8b39-9fd9d1efe743 2025-08-04T14:58:05.375657Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3742s 2025-08-04T14:58:05.376747Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:05.475768Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:05.562646Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71f44e74d5-3187-4468-a198-b329fdc367be 2025-08-04T14:58:05.562835Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5614s 2025-08-04T14:58:05.563830Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:05.638170Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.05s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.05s) === RUN TestTrustDomainValidation/From_primary-0 2025-08-04T14:56:31.422204Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T14:58:06.048402Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m34.6261904s)' === 2025-08-04T14:58:06.048533Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T14:58:06.048544Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T14:58:06.048534Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T14:58:06.123859Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/_test_context/istio-state162675280/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:06.130455Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:06.180346Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/_test_context/istio-state162675280/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:06.181111Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:06.188038Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:06.288374Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71ba8a6307-9971-441f-87f5-cdb0d055b1f6 2025-08-04T14:58:06.288818Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2401s 2025-08-04T14:58:06.289214Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:06.673344Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/711b9dcb28-03f7-4e69-9044-93518a30ceda 2025-08-04T14:58:06.673618Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6248s 2025-08-04T14:58:06.674479Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:06.767182Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:58:06.885809Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71d5ed1cd8-1e19-4114-add9-b3aca7491888 2025-08-04T14:58:06.885953Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8372s 2025-08-04T14:58:06.886852Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF --- FAIL: TestTrustDomainValidation/From_primary-0 (95.91s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (95.91s) === RUN TestTrustDomainValidation 2025-08-04T14:56:31.404407Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T14:58:07.336102Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m35.931687503s)' === 2025-08-04T14:58:07.336202Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T14:58:07.336203Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T14:58:07.336211Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T14:58:07.385621Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/_test_context/istio-state3482874514/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:07.393477Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:07.423297Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/_test_context/istio-state3482874514/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:07.430873Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:07.493431Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:07.701785Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71e9f59d3d-3fb8-41f4-a970-b6b0a7a9a7ae 2025-08-04T14:58:07.702277Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3659s 2025-08-04T14:58:07.702808Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:07.836153Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7172b42fb2-7f9d-473c-bca9-d6b6422946cc 2025-08-04T14:58:07.836383Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5000s 2025-08-04T14:58:07.837172Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:07.945930Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:58:07.996709Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:07.996802Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:08.002977Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71c9d5aa0b-ae60-4236-977d-d236dd4f0571 2025-08-04T14:58:08.003091Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6667s 2025-08-04T14:58:08.003755Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:08.296255Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainValidation (97.14s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (97.14s) 2025-08-04T14:58:08.541686Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-08-04T14:58:08.541764Z info tf === Suite "security_ca_custom_root" run time: 2m50.649057309s === 2025-08-04T14:58:08.541779Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-08-04T14:58:08.542015Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:08.542033Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:08.542037Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:08.570514Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/_suite_context/istio-state-3817213365/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:08.585764Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:08.633398Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/_suite_context/istio-state-3817213365/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:08.640329Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:08.921667Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:08Z/72c18493c7-5685-47ba-9630-77a4b04a40af 2025-08-04T14:58:08.922013Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3798s 2025-08-04T14:58:08.922855Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:09.022388Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:08Z/7275f21c96-45b6-4b01-9042-8eb866431847 2025-08-04T14:58:09.022490Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4802s 2025-08-04T14:58:09.023184Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:09.124044Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:08Z/72857c9611-0015-407a-a586-06a760d1a398 2025-08-04T14:58:09.124171Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5819s 2025-08-04T14:58:09.124749Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:09.125116Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:09.301651Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:58:09.480126Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:09.722236Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T14:58:09.722335Z info tf === SUCCEEDED: Cleanup Istio in 90.035µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root DONE 22 tests, 9 failures in 182.428s exec: [go test -json -test.run=^TestSecureNaming$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 470086 2025-08-04T14:58:21.130143Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:58:21.130212Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:58:21.157802Z info tf === Test Framework Settings === 2025-08-04T14:58:21.157889Z info tf TestID: security_ca_custom_root RunID: 7ab6ae2a-7dee-451a-927a-52b87a5e674e NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T14:58:21.157894Z info tf =============================== 2025-08-04T14:58:21.158104Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2 2025-08-04T14:58:21.158175Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T14:58:21.158184Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T14:58:21.158189Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T14:58:21.158203Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T14:58:21.158210Z info tf === BEGIN: Building clusters === 2025-08-04T14:58:21.159569Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T14:58:21.159576Z info tf === DONE: Building clusters === 2025-08-04T14:58:21.159588Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T14:58:21.194267Z info configmap istio-ca-root-cert is deleted 2025-08-04T14:58:21.194336Z info tf === SUCCESS: context setup function === 2025-08-04T14:58:21.194346Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T14:58:21.194364Z info tf === Istio Component Config === 2025-08-04T14:58:21.194429Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T14:58:21.194436Z info tf ================================ 2025-08-04T14:58:21.197541Z info tf skipping deployment as specified in the config 2025-08-04T14:58:21.197562Z info tf === SUCCEEDED: Deploy Istio in 3.209708ms [Suite=security_ca_custom_root]=== 2025-08-04T14:58:21.197572Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T14:58:21.274128Z info tf === SUCCEEDED: Create namespace echo1 in 76.548545ms === Generating RSA private key, 2048 bit long modulus (2 primes) ************************************+++++ ***+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/ca-custom-root-2901045714/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *****************************************************************************************************+++++ ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/ca-custom-root-2901045714/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/ca-custom-root-2901045714/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T14:58:22.879681Z info tf === BEGIN: Create namespace external === 2025-08-04T14:58:22.887846Z info tf === SUCCEEDED: Create namespace external in 8.179639ms === 2025-08-04T14:58:22.951060Z info tf === BEGIN: Deploy echo instances === 2025-08-04T14:58:22.967442Z error tf === FAILED: Deploy echo instances === 2025-08-04T14:58:22.967492Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/Service.2431944718.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T14:58:22.967528Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:22.967539Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:22.967551Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:23.035692Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/istio-state-2319868730/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:23.051601Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:23.081362Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/istio-state-2319868730/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:23.086640Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:23.275003Z error klog an error occurred forwarding 40097 -> 15000: error forwarding port 15000 to pod c9d8a06ca832f45ca1e9440dab9d86e07e9558d9c2cfc8ec75d4e87691cb27dd, uid : port forward into network namespace "/var/run/netns/5b5d06a7-e337-4d6c-af48-eec5686366c7": failed to connect to localhost:15000 inside namespace c9d8a06ca832f45ca1e9440dab9d86e07e9558d9c2cfc8ec75d4e87691cb27dd: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.276570Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:40097/config_dump?include_eds=true": EOF 2025-08-04T14:58:23.276630Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.277243Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:40097/clusters": dial tcp 127.0.0.1:40097: connect: connection refused 2025-08-04T14:58:23.307302Z error klog an error occurred forwarding 35353 -> 15020: error forwarding port 15020 to pod 6b393c155560294f858585f09cac4d069dde1e747094131ebac89b1a5c4472cb, uid : port forward into network namespace "/var/run/netns/b524eba7-edc7-49fe-8679-f636e774fc4f": failed to connect to localhost:15020 inside namespace 6b393c155560294f858585f09cac4d069dde1e747094131ebac89b1a5c4472cb: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.308996Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.309025Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-gdxgm for: Get "http://localhost:35353/debug/ndsz": EOF 2025-08-04T14:58:23.309722Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-gdxgm for: Get "http://localhost:35353/stats/prometheus": dial tcp 127.0.0.1:35353: connect: connection refused 2025-08-04T14:58:23.316594Z error klog an error occurred forwarding 45239 -> 15020: error forwarding port 15020 to pod a87f66a61cf970db97f44188cb3680bc0d7afb08763f467dfb7253555d5961ec, uid : port forward into network namespace "/var/run/netns/0f62e778-7363-4187-8698-b81bc6b9bc43": failed to connect to localhost:15020 inside namespace a87f66a61cf970db97f44188cb3680bc0d7afb08763f467dfb7253555d5961ec: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.318479Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qwk6x for: Get "http://localhost:45239/debug/ndsz": EOF 2025-08-04T14:58:23.318530Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.319140Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qwk6x for: Get "http://localhost:45239/stats/prometheus": dial tcp 127.0.0.1:45239: connect: connection refused 2025-08-04T14:58:23.330641Z error klog an error occurred forwarding 46225 -> 15000: error forwarding port 15000 to pod 03abc71c503efe866940150e487efb134412fd4a1e9893338aed3e7218ff2f78, uid : port forward into network namespace "/var/run/netns/51b54a91-1912-47d2-b4c5-9ff54aa07096": failed to connect to localhost:15000 inside namespace 03abc71c503efe866940150e487efb134412fd4a1e9893338aed3e7218ff2f78: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.332337Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-8mkqg for: Get "http://localhost:46225/config_dump?include_eds=true": EOF 2025-08-04T14:58:23.332407Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.333022Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-8mkqg for: Get "http://localhost:46225/clusters": dial tcp 127.0.0.1:46225: connect: connection refused 2025-08-04T14:58:23.353878Z error klog an error occurred forwarding 37403 -> 15000: error forwarding port 15000 to pod 454c744a16dd73ff08ed483861d69efd7ad3d23a7d13af5ca6673a186ed4fb57, uid : port forward into network namespace "/var/run/netns/81245c18-d285-42cc-a197-76726ff6b017": failed to connect to localhost:15000 inside namespace 454c744a16dd73ff08ed483861d69efd7ad3d23a7d13af5ca6673a186ed4fb57: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.356003Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-dcnbn for: Get "http://localhost:37403/config_dump?include_eds=true": EOF 2025-08-04T14:58:23.356036Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.358387Z error klog an error occurred forwarding 45645 -> 15000: error forwarding port 15000 to pod a87f66a61cf970db97f44188cb3680bc0d7afb08763f467dfb7253555d5961ec, uid : port forward into network namespace "/var/run/netns/0f62e778-7363-4187-8698-b81bc6b9bc43": failed to connect to localhost:15000 inside namespace a87f66a61cf970db97f44188cb3680bc0d7afb08763f467dfb7253555d5961ec: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.358564Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-vkfk5_echo1") 2025-08-04T14:58:23.358932Z error klog an error occurred forwarding 35493 -> 15020: error forwarding port 15020 to pod 03abc71c503efe866940150e487efb134412fd4a1e9893338aed3e7218ff2f78, uid : port forward into network namespace "/var/run/netns/51b54a91-1912-47d2-b4c5-9ff54aa07096": failed to connect to localhost:15020 inside namespace 03abc71c503efe866940150e487efb134412fd4a1e9893338aed3e7218ff2f78: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.360783Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qwk6x for: Get "http://localhost:45645/config_dump?include_eds=true": EOF 2025-08-04T14:58:23.360828Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-8mkqg for: Get "http://localhost:35493/debug/ndsz": EOF 2025-08-04T14:58:23.360891Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.360895Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.374617Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:23.374929Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-jg696_echo1") 2025-08-04T14:58:23.374929Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-f6f5f8cb7-njg2s_echo1") 2025-08-04T14:58:23.412674Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qwk6x for: Get "http://localhost:45645/clusters": dial tcp 127.0.0.1:45645: connect: connection refused 2025-08-04T14:58:23.412780Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-dcnbn for: Get "http://localhost:37403/clusters": dial tcp 127.0.0.1:37403: connect: connection refused 2025-08-04T14:58:23.412644Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-8mkqg for: Get "http://localhost:35493/stats/prometheus": dial tcp 127.0.0.1:35493: connect: connection refused 2025-08-04T14:58:23.427611Z error klog an error occurred forwarding 38257 -> 15020: error forwarding port 15020 to pod 454c744a16dd73ff08ed483861d69efd7ad3d23a7d13af5ca6673a186ed4fb57, uid : port forward into network namespace "/var/run/netns/81245c18-d285-42cc-a197-76726ff6b017": failed to connect to localhost:15020 inside namespace 454c744a16dd73ff08ed483861d69efd7ad3d23a7d13af5ca6673a186ed4fb57: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.429161Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-dcnbn for: Get "http://localhost:38257/debug/ndsz": EOF 2025-08-04T14:58:23.429324Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.430112Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-dcnbn for: Get "http://localhost:38257/stats/prometheus": dial tcp 127.0.0.1:38257: connect: connection refused 2025-08-04T14:58:23.477345Z error klog an error occurred forwarding 36769 -> 15000: error forwarding port 15000 to pod 6b393c155560294f858585f09cac4d069dde1e747094131ebac89b1a5c4472cb, uid : port forward into network namespace "/var/run/netns/b524eba7-edc7-49fe-8679-f636e774fc4f": failed to connect to localhost:15000 inside namespace 6b393c155560294f858585f09cac4d069dde1e747094131ebac89b1a5c4472cb: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.479627Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-gdxgm for: Get "http://localhost:36769/config_dump?include_eds=true": EOF 2025-08-04T14:58:23.479702Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.480324Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-gdxgm for: Get "http://localhost:36769/clusters": dial tcp 127.0.0.1:36769: connect: connection refused 2025-08-04T14:58:23.480977Z error klog an error occurred forwarding 44029 -> 15000: error forwarding port 15000 to pod c936f072ce6b86a8b682931a35d63f2164395724656849dd8f05f7dbc9f615c2, uid : port forward into network namespace "/var/run/netns/cd072a75-1b8a-4d1d-9784-04f32afa8f14": failed to connect to localhost:15000 inside namespace c936f072ce6b86a8b682931a35d63f2164395724656849dd8f05f7dbc9f615c2: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:23.483394Z error port forward failed: lost connection to pod 2025-08-04T14:58:23.483391Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-rh6lc for: Get "http://localhost:44029/config_dump?include_eds=true": EOF 2025-08-04T14:58:23.484146Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-rh6lc for: Get "http://localhost:44029/clusters": dial tcp 127.0.0.1:44029: connect: connection refused 2025-08-04T14:58:23.848596Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:58:24.043774Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:24.340806Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:24.341568Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-jg696_echo1") 2025-08-04T14:58:24.343844Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-f6f5f8cb7-njg2s_echo1") 2025-08-04T14:58:24.388203Z error klog an error occurred forwarding 38941 -> 15000: error forwarding port 15000 to pod bfa1c9a1e45df286dba004086005a35da826369e49ed6effc0a41b3d45b0e181, uid : port forward into network namespace "/var/run/netns/cba04539-7875-4083-8c3e-746eae6abffa": failed to connect to localhost:15000 inside namespace bfa1c9a1e45df286dba004086005a35da826369e49ed6effc0a41b3d45b0e181: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:24.388720Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-vkfk5_echo1") 2025-08-04T14:58:24.390504Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-xbjp4 for: Get "http://localhost:38941/config_dump?include_eds=true": EOF 2025-08-04T14:58:24.390547Z error port forward failed: lost connection to pod 2025-08-04T14:58:24.391443Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-xbjp4 for: Get "http://localhost:38941/clusters": dial tcp 127.0.0.1:38941: connect: connection refused 2025-08-04T14:58:24.905040Z error klog an error occurred forwarding 35109 -> 15020: error forwarding port 15020 to pod c9d8a06ca832f45ca1e9440dab9d86e07e9558d9c2cfc8ec75d4e87691cb27dd, uid : port forward into network namespace "/var/run/netns/5b5d06a7-e337-4d6c-af48-eec5686366c7": failed to connect to localhost:15020 inside namespace c9d8a06ca832f45ca1e9440dab9d86e07e9558d9c2cfc8ec75d4e87691cb27dd: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T14:58:24.907054Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:35109/debug/ndsz": EOF 2025-08-04T14:58:24.907089Z error port forward failed: lost connection to pod 2025-08-04T14:58:24.908150Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:35109/stats/prometheus": dial tcp 127.0.0.1:35109: connect: connection refused 2025-08-04T14:58:25.003375Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("external-v1-56749b6f5c-zhzlh_external") 2025-08-04T14:58:25.007643Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-xbjp4_echo1") 2025-08-04T14:58:25.012063Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-jg696_echo1") 2025-08-04T14:58:25.012083Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-f6f5f8cb7-njg2s_echo1") 2025-08-04T14:58:25.012195Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.012691Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-gzs6l_echo1") 2025-08-04T14:58:25.065223Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.065490Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-vkfk5_echo1") 2025-08-04T14:58:25.071671Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-xwfmm_echo1") 2025-08-04T14:58:25.071814Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:25.072099Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:18Z/785e5fc95c-b6eb-4704-aa3e-57056c598597 2025-08-04T14:58:25.072353Z info tf istioctl ([x internal-debug --all configz]): completed after 2.1044s 2025-08-04T14:58:25.072937Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:25.097338Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-f6f5f8cb7-njg2s_echo1") 2025-08-04T14:58:25.105414Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-gzs6l_echo1") 2025-08-04T14:58:25.105912Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.108347Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-jg696_echo1") 2025-08-04T14:58:25.110386Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("external-v1-56749b6f5c-zhzlh_external") 2025-08-04T14:58:25.111556Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-xbjp4_echo1") 2025-08-04T14:58:25.151536Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.159199Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-xwfmm_echo1") 2025-08-04T14:58:25.165808Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:18Z/781bd7a58f-eb62-43ac-9b88-192dd343c49a 2025-08-04T14:58:25.165877Z info tf istioctl ([x internal-debug --all mcsz]): completed after 2.1979s 2025-08-04T14:58:25.166493Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:25.177151Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:25.181276Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-vkfk5_echo1") 2025-08-04T14:58:25.203042Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-f6f5f8cb7-njg2s_echo1") 2025-08-04T14:58:25.207295Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-xbjp4_echo1") 2025-08-04T14:58:25.210609Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.211433Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-gzs6l_echo1") 2025-08-04T14:58:25.223986Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-jg696_echo1") 2025-08-04T14:58:25.259744Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("external-v1-56749b6f5c-zhzlh_external") 2025-08-04T14:58:25.266435Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.276570Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1")) 2025-08-04T14:58:25.286192Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-xwfmm_echo1") 2025-08-04T14:58:25.296614Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:18Z/7853addb87-ac64-4659-8846-881c27251253 2025-08-04T14:58:25.296682Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.3287s 2025-08-04T14:58:25.297262Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:25.310193Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-vkfk5_echo1") 2025-08-04T14:58:25.315040Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.318742Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-gzs6l_echo1") 2025-08-04T14:58:25.322378Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:25.322817Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-xbjp4_echo1") 2025-08-04T14:58:25.341119Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("external-v1-56749b6f5c-zhzlh_external") 2025-08-04T14:58:25.413225Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:25.418594Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-xbjp4_echo1") 2025-08-04T14:58:25.423893Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-xwfmm_echo1") 2025-08-04T14:58:25.433785Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.462770Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("external-v1-56749b6f5c-zhzlh_external") 2025-08-04T14:58:25.473299Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("external-v1-56749b6f5c-zhzlh_external")) 2025-08-04T14:58:25.484590Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:25.505695Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:25.531647Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-xwfmm_echo1") 2025-08-04T14:58:25.542300Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-f6f5f8cb7-njg2s_echo1")) 2025-08-04T14:58:25.577218Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/Service.2431944718.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T14:58:25.577242Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/Service.2431944718.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-08-04T14:58:25.577248Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/Service.2431944718.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T14:58:25.577272Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:25.577286Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:25.577307Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:25.683318Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T14:58:25.714141Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/istio-state-1908730125/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:25.726088Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:25.858773Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-c56579cdb-gdxgm_echo1") 2025-08-04T14:58:25.873497Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-c56579cdb-gdxgm_echo1") 2025-08-04T14:58:25.875054Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("client-v1-b45b6bf85-dcnbn_echo1") 2025-08-04T14:58:25.888425Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.892053Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7ab6ae2/_suite_context/istio-state-1908730125/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:25.932208Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:25.941683Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:25.941689Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-nk79v_echo1") 2025-08-04T14:58:25.943319Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-xbjp4_echo1") 2025-08-04T14:58:25.944428Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:18Z/782aeedcd7-13d0-43b3-a3f6-b9562166cccc 2025-08-04T14:58:25.944670Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3671s 2025-08-04T14:58:25.944789Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:25.945234Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:25.947975Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-08-04T14:58:25.955373Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-8mkqg_echo1") 2025-08-04T14:58:25.955404Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-xwfmm_echo1") 2025-08-04T14:58:25.979145Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-08-04T14:58:26.084915Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:26.085042Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-08-04T14:58:26.091460Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:26.093000Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:18Z/78e8fc6f89-b44e-44bb-9d93-1d9193e6d8d2 2025-08-04T14:58:26.093095Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5155s 2025-08-04T14:58:26.093775Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:26.126862Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-08-04T14:58:26.136586Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-8mkqg_echo1") 2025-08-04T14:58:26.150500Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-8mkqg_echo1") 2025-08-04T14:58:26.194376Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:26.203728Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-v1-7cd6ccbd65-rh6lc_echo1") 2025-08-04T14:58:26.204376Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:58:18Z/78000cba98-0c5c-40e0-9f1e-46eeaf24e1cc 2025-08-04T14:58:26.204466Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6268s 2025-08-04T14:58:26.205117Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:26.244905Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T14:58:26.254212Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-b45b6bf85-dcnbn/app for: pods "client-v1-b45b6bf85-dcnbn" not found 2025-08-04T14:58:26.256418Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-b45b6bf85-dcnbn/istio-proxy for: pods "client-v1-b45b6bf85-dcnbn" not found 2025-08-04T14:58:26.258332Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/client-v1-b45b6bf85-dcnbn/istio-validation for: pods "client-v1-b45b6bf85-dcnbn" not found 2025-08-04T14:58:26.271829Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-xbjp4_echo1") 2025-08-04T14:58:26.431344Z error port forward failed: error upgrading connection: pods "client-v1-b45b6bf85-dcnbn" not found 2025-08-04T14:58:26.491334Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-6f49f7bdcb-xwfmm/istio-proxy for: pods "proxyless-grpc-v1-6f49f7bdcb-xwfmm" not found 2025-08-04T14:58:26.493907Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-6f49f7bdcb-xwfmm/app for: pods "proxyless-grpc-v1-6f49f7bdcb-xwfmm" not found 2025-08-04T14:58:26.496039Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-bar-v1-dbb69ff9f-gzs6l/app for: pods "server-naked-bar-v1-dbb69ff9f-gzs6l" not found 2025-08-04T14:58:26.497894Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T14:58:26.497996Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-alt-v1-645b4f7b87-jg696/app for: pods "server-naked-foo-alt-v1-645b4f7b87-jg696" not found 2025-08-04T14:58:26.500247Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-v1-f6f5f8cb7-njg2s/app for: pods "server-naked-foo-v1-f6f5f8cb7-njg2s" not found 2025-08-04T14:58:26.501250Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:58:26.502222Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-v1-7cd6ccbd65-rh6lc/app for: pods "server-v1-7cd6ccbd65-rh6lc" not found 2025-08-04T14:58:26.504327Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-v1-7cd6ccbd65-rh6lc/istio-proxy for: pods "server-v1-7cd6ccbd65-rh6lc" not found 2025-08-04T14:58:26.506177Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-v1-7cd6ccbd65-rh6lc/istio-validation for: pods "server-v1-7cd6ccbd65-rh6lc" not found 2025-08-04T14:58:26.507989Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-867646b8b5-xbjp4/app for: pods "sotw-v1-867646b8b5-xbjp4" not found 2025-08-04T14:58:26.509708Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-867646b8b5-xbjp4/istio-proxy for: pods "sotw-v1-867646b8b5-xbjp4" not found 2025-08-04T14:58:26.511884Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/sotw-v1-867646b8b5-xbjp4/istio-validation for: pods "sotw-v1-867646b8b5-xbjp4" not found 2025-08-04T14:58:26.514115Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/app for: pods "statefulset-v1-0" not found 2025-08-04T14:58:26.515950Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-proxy for: pods "statefulset-v1-0" not found 2025-08-04T14:58:26.517663Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-validation for: pods "statefulset-v1-0" not found 2025-08-04T14:58:26.520047Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T14:58:26.520076Z info tf === SUCCEEDED: Cleanup Istio in 23.611µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0/server:server-naked-foo\[success\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 471507 2025-08-04T14:58:37.708435Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:58:37.708505Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:58:37.739468Z info tf === Test Framework Settings === 2025-08-04T14:58:37.739595Z info tf TestID: security_ca_custom_root RunID: e2d58b7b-f842-42d6-a3fa-f22069a98b93 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T14:58:37.739606Z info tf =============================== 2025-08-04T14:58:37.739781Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7 2025-08-04T14:58:37.739893Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T14:58:37.739908Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T14:58:37.739917Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T14:58:37.739935Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T14:58:37.739943Z info tf === BEGIN: Building clusters === 2025-08-04T14:58:37.741561Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T14:58:37.741570Z info tf === DONE: Building clusters === 2025-08-04T14:58:37.741588Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T14:58:37.791122Z info configmap istio-ca-root-cert is deleted 2025-08-04T14:58:37.791210Z info tf === SUCCESS: context setup function === 2025-08-04T14:58:37.791220Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T14:58:37.791241Z info tf === Istio Component Config === 2025-08-04T14:58:37.791282Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T14:58:37.791332Z info tf ================================ 2025-08-04T14:58:37.795764Z info tf skipping deployment as specified in the config 2025-08-04T14:58:37.795798Z info tf === SUCCEEDED: Deploy Istio in 4.568701ms [Suite=security_ca_custom_root]=== 2025-08-04T14:58:37.795808Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T14:58:37.859947Z info tf === SUCCEEDED: Create namespace echo1 in 64.128052ms === Generating RSA private key, 2048 bit long modulus (2 primes) ******************************************************************************************************************************************************************************+++++ ********************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7/_suite_context/ca-custom-root-112817494/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ******************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7/_suite_context/ca-custom-root-112817494/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7/_suite_context/ca-custom-root-112817494/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T14:58:39.349096Z info tf === BEGIN: Create namespace external === 2025-08-04T14:58:39.356519Z info tf === SUCCEEDED: Create namespace external in 7.45441ms === 2025-08-04T14:58:39.410609Z info tf === BEGIN: Deploy echo instances === 2025-08-04T14:58:39.425778Z error tf === FAILED: Deploy echo instances === 2025-08-04T14:58:39.425844Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7/_suite_context/Service.865063787.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T14:58:39.425883Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:39.425888Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:39.425923Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:39.434363Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7/_suite_context/Service.865063787.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T14:58:39.434433Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7/_suite_context/Service.865063787.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-08-04T14:58:39.434441Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-e2d58b7/_suite_context/Service.865063787.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T14:58:39.434484Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:39.434512Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:39.434527Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T14:58:39.453981Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T14:58:39.454083Z info tf === SUCCEEDED: Cleanup Istio in 94.355µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 472877 2025-08-04T14:58:51.619997Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:58:51.620075Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T14:58:51.654262Z info tf === Test Framework Settings === 2025-08-04T14:58:51.654397Z info tf TestID: security_ca_custom_root RunID: 5f1cb773-c440-44e8-820b-d4b93ac3c0b6 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T14:58:51.654407Z info tf =============================== 2025-08-04T14:58:51.654650Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77 2025-08-04T14:58:51.654757Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T14:58:51.654779Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T14:58:51.654789Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T14:58:51.654811Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T14:58:51.654824Z info tf === BEGIN: Building clusters === 2025-08-04T14:58:51.656701Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T14:58:51.656710Z info tf === DONE: Building clusters === 2025-08-04T14:58:51.656730Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T14:58:51.695792Z info configmap istio-ca-root-cert is deleted 2025-08-04T14:58:51.695842Z info tf === SUCCESS: context setup function === 2025-08-04T14:58:51.695856Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T14:58:51.695881Z info tf === Istio Component Config === 2025-08-04T14:58:51.695960Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T14:58:51.695970Z info tf ================================ 2025-08-04T14:58:51.700679Z info tf skipping deployment as specified in the config 2025-08-04T14:58:51.700707Z info tf === SUCCEEDED: Deploy Istio in 4.844359ms [Suite=security_ca_custom_root]=== 2025-08-04T14:58:51.700715Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T14:58:51.858997Z info tf === SUCCEEDED: Create namespace echo1 in 158.262709ms === Generating RSA private key, 2048 bit long modulus (2 primes) ******************************************************************************************************************************************************************************+++++ *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/_suite_context/ca-custom-root-3668179186/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) **************************************************************************************************************************+++++ ***********************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/_suite_context/ca-custom-root-3668179186/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *******************************************************************************************************************************************************************************************************************************************+++++ **************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/_suite_context/ca-custom-root-3668179186/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T14:58:53.000809Z info tf === BEGIN: Create namespace external === 2025-08-04T14:58:53.255368Z info tf === SUCCEEDED: Create namespace external in 254.583611ms === 2025-08-04T14:58:53.338158Z info tf === BEGIN: Deploy echo instances === 2025-08-04T14:58:53.698838Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.724402Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.728145Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.742069Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.750468Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.750635Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.753070Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.771343Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.771357Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.799456Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.806814Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.815256Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.824604Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.829225Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.843027Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.850955Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.850919Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.853770Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.854639Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.854644Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.863121Z info klog Waiting for caches to sync controller=echo 2025-08-04T14:58:53.871784Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.871910Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.907031Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.915736Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.955009Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.955009Z info klog Caches are synced controller=echo 2025-08-04T14:58:53.963908Z info klog Caches are synced controller=echo 2025-08-04T14:59:16.273864Z info tf === SUCCEEDED: Deploy echo instances in 22.935699838s === 2025-08-04T14:59:16.303870Z info tf === DONE: Setup: 'security_ca_custom_root' (24.647122737s) === 2025-08-04T14:59:16.303910Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] 2025-08-04T14:59:16.321052Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.0.195:34416->172.30.142.25:443: read: connection reset by peer 2025-08-04T14:59:46.327289Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006227919s)' === 2025-08-04T14:59:46.327446Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:59:46.327458Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:59:46.327470Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:59:46.402559Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state3194475387/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:59:46.452973Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:59:46.529776Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state3194475387/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:59:46.556957Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:59:46.786132Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86806909ce-3b88-4b27-9b2b-884b9b046610 2025-08-04T14:59:46.786772Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4589s 2025-08-04T14:59:46.787080Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:46.943211Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86af36d6da-5880-40c3-bdfd-a3f9e16abb0e 2025-08-04T14:59:46.943374Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6155s 2025-08-04T14:59:46.944256Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:47.006373Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:47.133120Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/869f0fe587-c350-4687-ad93-9a0ff682718c 2025-08-04T14:59:47.133290Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.8054s 2025-08-04T14:59:47.134305Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:47.271353Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:59:47.418331Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (31.36s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 1) (31.36s) PASS tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-bar[fail] (re-run 1) (0.03s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0 2025-08-04T14:59:16.320814Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T14:59:47.709238Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (31.388422928s)' === 2025-08-04T14:59:47.709292Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:59:47.709301Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:59:47.709297Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:59:47.741244Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1578390482/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:59:47.758784Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:59:47.790417Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1578390482/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:59:47.796113Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:59:47.892537Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/866b360e55-a11b-407e-a9e1-4b6db1fde31b 2025-08-04T14:59:47.892885Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1831s 2025-08-04T14:59:47.893607Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.004135Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86aad00239-e154-4b2c-b0b7-a23f0d32a528 2025-08-04T14:59:48.004289Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2945s 2025-08-04T14:59:48.005170Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.104623Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86e5bd0c12-f5ea-4922-9bf3-364fd38b4a4d 2025-08-04T14:59:48.104773Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3950s 2025-08-04T14:59:48.105433Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.142951Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:48.471786Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:59:48.474202Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0 (32.27s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 1) (32.27s) === RUN TestTrustDomainAliasSecureNaming 2025-08-04T14:59:16.304388Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T14:59:48.588530Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.28413147s)' === 2025-08-04T14:59:48.588652Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:59:48.588654Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:59:48.588688Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:59:48.634967Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/_test_context/istio-state2149072087/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:59:48.650092Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:59:48.668811Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/_test_context/istio-state2149072087/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:59:48.674454Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:59:48.812334Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86e021a553-d3fb-431c-aaa3-3a8a0b16d24f 2025-08-04T14:59:48.812813Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2239s 2025-08-04T14:59:48.813437Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.939359Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/865db2fe4b-0f2f-4506-ae1b-108209712063 2025-08-04T14:59:48.939490Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3506s 2025-08-04T14:59:48.940491Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:49.076605Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:49.076594Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/869fc9d828-2556-48cf-aaf5-50b68640c6da 2025-08-04T14:59:49.076946Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4880s 2025-08-04T14:59:49.077789Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:49.255378Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:59:49.411732Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming (33.21s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming (re-run 1) (33.21s) 2025-08-04T14:59:49.512780Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-08-04T14:59:49.512863Z info tf === Suite "security_ca_custom_root" run time: 57.856132552s === 2025-08-04T14:59:49.512881Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-08-04T14:59:49.513319Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T14:59:49.513330Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T14:59:49.513341Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T14:59:49.540820Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/_suite_context/istio-state-1551771228/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:59:49.561869Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:59:49.597499Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/_suite_context/istio-state-1551771228/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:59:49.603396Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:59:49.985498Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:49Z/87c633460b-fd7c-4abb-9691-b7cd19b073a7 2025-08-04T14:59:49.985847Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4720s 2025-08-04T14:59:49.986666Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:50.143528Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:49Z/8778be1a8d-6e42-4de1-b451-f060c587da7c 2025-08-04T14:59:50.143657Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6298s 2025-08-04T14:59:50.144401Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:50.265327Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:50.320304Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:49Z/87c8e3d38f-a354-4554-9064-ff228640bffe 2025-08-04T14:59:50.320483Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.8066s 2025-08-04T14:59:50.321146Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:50.547306Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:59:50.858850Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:51.511980Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T14:59:51.512062Z info tf === SUCCEEDED: Cleanup Istio in 76.362µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:http-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 474470 2025-08-04T15:00:03.244045Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:03.244097Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:03.272373Z info tf === Test Framework Settings === 2025-08-04T15:00:03.272457Z info tf TestID: security_ca_custom_root RunID: 8994d98d-b5ba-4b08-adb1-bcfa59268bad NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:00:03.272462Z info tf =============================== 2025-08-04T15:00:03.274678Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98 2025-08-04T15:00:03.275581Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:00:03.275592Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:00:03.275598Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:00:03.276414Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:00:03.276423Z info tf === BEGIN: Building clusters === 2025-08-04T15:00:03.277969Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:00:03.277976Z info tf === DONE: Building clusters === 2025-08-04T15:00:03.277991Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:00:03.334161Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:00:03.334309Z info tf === SUCCESS: context setup function === 2025-08-04T15:00:03.334318Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:00:03.334330Z info tf === Istio Component Config === 2025-08-04T15:00:03.334377Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:00:03.334383Z info tf ================================ 2025-08-04T15:00:03.338638Z info tf skipping deployment as specified in the config 2025-08-04T15:00:03.338653Z info tf === SUCCEEDED: Deploy Istio in 4.33103ms [Suite=security_ca_custom_root]=== 2025-08-04T15:00:03.338660Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:00:03.397613Z info tf === SUCCEEDED: Create namespace echo1 in 58.947804ms === Generating RSA private key, 2048 bit long modulus (2 primes) ************************+++++ ****************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/ca-custom-root-792131335/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *****************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/ca-custom-root-792131335/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************************************************************************+++++ ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/ca-custom-root-792131335/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:00:04.769086Z info tf === BEGIN: Create namespace external === 2025-08-04T15:00:04.792317Z info tf === SUCCEEDED: Create namespace external in 23.242113ms === 2025-08-04T15:00:04.864149Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:00:04.888008Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:00:04.888025Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/Service.533372912.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:00:04.888057Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:04.888067Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:04.888084Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:04.954667Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/istio-state-337755292/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:00:05.002570Z error klog an error occurred forwarding 36473 -> 15020: error forwarding port 15020 to pod 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c, uid : port forward into network namespace "/var/run/netns/94f3dc9b-e633-4bdc-8ecb-b2e0fcb5f6fe": failed to connect to localhost:15020 inside namespace 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:05.004539Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:36473/debug/ndsz": EOF 2025-08-04T15:00:05.004609Z error port forward failed: lost connection to pod 2025-08-04T15:00:05.005462Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:36473/stats/prometheus": dial tcp 127.0.0.1:36473: connect: connection refused 2025-08-04T15:00:05.031633Z error klog an error occurred forwarding 43019 -> 15020: error forwarding port 15020 to pod b35c0cbe02d0b57cca837dcd55e8da0433d5335f3a9ea1bab90bec6589bee743, uid : port forward into network namespace "/var/run/netns/9b9f4865-db86-4fef-8bcd-e0eefbd49c07": failed to connect to localhost:15020 inside namespace b35c0cbe02d0b57cca837dcd55e8da0433d5335f3a9ea1bab90bec6589bee743: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:05.032987Z error klog an error occurred forwarding 36475 -> 15000: error forwarding port 15000 to pod 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26, uid : port forward into network namespace "/var/run/netns/a590261b-e5db-442b-97a4-bcdfa2d1c415": failed to connect to localhost:15000 inside namespace 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:05.033424Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43019/debug/ndsz": EOF 2025-08-04T15:00:05.033539Z error port forward failed: lost connection to pod 2025-08-04T15:00:05.034553Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43019/stats/prometheus": dial tcp 127.0.0.1:43019: connect: connection refused 2025-08-04T15:00:05.034625Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:36475/config_dump?include_eds=true": EOF 2025-08-04T15:00:05.034730Z error port forward failed: lost connection to pod 2025-08-04T15:00:05.035201Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:36475/clusters": dial tcp 127.0.0.1:36475: connect: connection refused 2025-08-04T15:00:05.186010Z error klog an error occurred forwarding 38647 -> 15020: error forwarding port 15020 to pod dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345, uid : port forward into network namespace "/var/run/netns/cef1f76f-ae58-47d3-8156-c86f834f48ad": failed to connect to localhost:15020 inside namespace dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:05.187854Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:38647/debug/ndsz": EOF 2025-08-04T15:00:05.187934Z error port forward failed: lost connection to pod 2025-08-04T15:00:05.188751Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:38647/stats/prometheus": dial tcp 127.0.0.1:38647: connect: connection refused 2025-08-04T15:00:06.456994Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:00:07.516359Z error klog an error occurred forwarding 39277 -> 15000: error forwarding port 15000 to pod bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01, uid : port forward into network namespace "/var/run/netns/e9103743-e437-40ee-9b4d-7289dc89e109": failed to connect to localhost:15000 inside namespace bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:07.518351Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:39277/config_dump?include_eds=true": EOF 2025-08-04T15:00:07.518423Z error port forward failed: lost connection to pod 2025-08-04T15:00:07.519160Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:39277/clusters": dial tcp 127.0.0.1:39277: connect: connection refused 2025-08-04T15:00:07.583435Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/istio-state-337755292/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:00:07.595409Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:00:07.820129Z error klog an error occurred forwarding 40961 -> 15020: error forwarding port 15020 to pod 0a2a0b770bf9d2549a10d4e565d0b5a1f6292243279982a180c931f686e2aa3f, uid : port forward into network namespace "/var/run/netns/41cc68c7-28c5-4345-b9f3-356ac618572b": failed to connect to localhost:15020 inside namespace 0a2a0b770bf9d2549a10d4e565d0b5a1f6292243279982a180c931f686e2aa3f: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:07.822339Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-88xt9 for: Get "http://localhost:40961/debug/ndsz": EOF 2025-08-04T15:00:07.822367Z error port forward failed: lost connection to pod 2025-08-04T15:00:07.823318Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-88xt9 for: Get "http://localhost:40961/stats/prometheus": dial tcp 127.0.0.1:40961: connect: connection refused 2025-08-04T15:00:07.997535Z error klog an error occurred forwarding 46825 -> 15000: error forwarding port 15000 to pod dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345, uid : port forward into network namespace "/var/run/netns/cef1f76f-ae58-47d3-8156-c86f834f48ad": failed to connect to localhost:15000 inside namespace dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:07.999885Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:46825/config_dump?include_eds=true": EOF 2025-08-04T15:00:07.999928Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.000876Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:46825/clusters": dial tcp 127.0.0.1:46825: connect: connection refused 2025-08-04T15:00:08.010132Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:59Z/90b7b3b3eb-37bf-4557-9d80-d586573ed97c 2025-08-04T15:00:08.010448Z info tf istioctl ([x internal-debug --all configz]): completed after 3.1212s 2025-08-04T15:00:08.011022Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:00:08.206432Z error klog an error occurred forwarding 37313 -> 15020: error forwarding port 15020 to pod bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01, uid : port forward into network namespace "/var/run/netns/e9103743-e437-40ee-9b4d-7289dc89e109": failed to connect to localhost:15020 inside namespace bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.208419Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:37313/debug/ndsz": EOF 2025-08-04T15:00:08.208437Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.209612Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:37313/stats/prometheus": dial tcp 127.0.0.1:37313: connect: connection refused 2025-08-04T15:00:08.224795Z error klog an error occurred forwarding 42673 -> 15020: error forwarding port 15020 to pod 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26, uid : port forward into network namespace "/var/run/netns/a590261b-e5db-442b-97a4-bcdfa2d1c415": failed to connect to localhost:15020 inside namespace 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.225189Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:00:08.226402Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.226403Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:42673/debug/ndsz": EOF 2025-08-04T15:00:08.227181Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:42673/stats/prometheus": dial tcp 127.0.0.1:42673: connect: connection refused 2025-08-04T15:00:08.276517Z error klog an error occurred forwarding 39587 -> 15000: error forwarding port 15000 to pod 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c, uid : port forward into network namespace "/var/run/netns/94f3dc9b-e633-4bdc-8ecb-b2e0fcb5f6fe": failed to connect to localhost:15000 inside namespace 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.278399Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:39587/config_dump?include_eds=true": EOF 2025-08-04T15:00:08.278462Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.279080Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:39587/clusters": dial tcp 127.0.0.1:39587: connect: connection refused 2025-08-04T15:00:08.305162Z error klog an error occurred forwarding 39303 -> 15000: error forwarding port 15000 to pod 0a2a0b770bf9d2549a10d4e565d0b5a1f6292243279982a180c931f686e2aa3f, uid : port forward into network namespace "/var/run/netns/41cc68c7-28c5-4345-b9f3-356ac618572b": failed to connect to localhost:15000 inside namespace 0a2a0b770bf9d2549a10d4e565d0b5a1f6292243279982a180c931f686e2aa3f: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.307059Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-88xt9 for: Get "http://localhost:39303/config_dump?include_eds=true": EOF 2025-08-04T15:00:08.307173Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.307982Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-88xt9 for: Get "http://localhost:39303/clusters": dial tcp 127.0.0.1:39303: connect: connection refused 2025-08-04T15:00:08.405735Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("external-v1-56749b6f5c-j25rs_external") 2025-08-04T15:00:08.410879Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-2scgg_echo1") 2025-08-04T15:00:08.412538Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-2hpv9_echo1") 2025-08-04T15:00:08.415967Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-2lwvg_echo1") 2025-08-04T15:00:08.445394Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-lgdr9_echo1") 2025-08-04T15:00:08.445472Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-wx5bg_echo1") 2025-08-04T15:00:08.445539Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-kmt7c_echo1") 2025-08-04T15:00:08.446334Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-kmt7c_echo1") 2025-08-04T15:00:08.447428Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:59Z/90d654c1e2-fd3b-4d47-914a-74f6ac7e4827 2025-08-04T15:00:08.447501Z info tf istioctl ([x internal-debug --all mcsz]): completed after 3.5583s 2025-08-04T15:00:08.448334Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:00:08.454321Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-08-04T15:00:08.454440Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-2hpv9_echo1") 2025-08-04T15:00:08.454447Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-v1-f6f5f8cb7-m56c7_echo1") 2025-08-04T15:00:08.490953Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:08.514470Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:00:08.617321Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-kmt7c_echo1") 2025-08-04T15:00:08.620780Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:59Z/90f2611765-980b-4834-85a4-a1da357cf917 2025-08-04T15:00:08.620852Z info tf istioctl ([x internal-debug --all clusterz]): completed after 3.7316s 2025-08-04T15:00:08.621714Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:00:08.669926Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:08.672169Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-kmt7c_echo1") 2025-08-04T15:00:08.726788Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:08.730016Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/Service.533372912.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:00:08.730057Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/Service.533372912.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-08-04T15:00:08.730066Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/Service.533372912.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:00:08.730113Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:08.730121Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:08.730148Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:08.821012Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/istio-state-858184591/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:00:08.837249Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:00:08.838003Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:08.886555Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8994d98/_suite_context/istio-state-858184591/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:00:08.894191Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:00:08.951195Z error klog an error occurred forwarding 33101 -> 15000: error forwarding port 15000 to pod 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26, uid : port forward into network namespace "/var/run/netns/a590261b-e5db-442b-97a4-bcdfa2d1c415": failed to connect to localhost:15000 inside namespace 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.953162Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:33101/config_dump?include_eds=true": EOF 2025-08-04T15:00:08.953343Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.954059Z error klog an error occurred forwarding 35623 -> 15000: error forwarding port 15000 to pod bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01, uid : port forward into network namespace "/var/run/netns/e9103743-e437-40ee-9b4d-7289dc89e109": failed to connect to localhost:15000 inside namespace bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.955806Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:35623/config_dump?include_eds=true": EOF 2025-08-04T15:00:08.955938Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.957147Z error klog an error occurred forwarding 33313 -> 15020: error forwarding port 15020 to pod dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345, uid : port forward into network namespace "/var/run/netns/cef1f76f-ae58-47d3-8156-c86f834f48ad": failed to connect to localhost:15020 inside namespace dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.958701Z error klog an error occurred forwarding 38523 -> 15000: error forwarding port 15000 to pod dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345, uid : port forward into network namespace "/var/run/netns/cef1f76f-ae58-47d3-8156-c86f834f48ad": failed to connect to localhost:15000 inside namespace dd2f61a80fc1ac7330c85955df36ebd51a8026c13aabdbbcca61c32accfd7345: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.959019Z error klog an error occurred forwarding 42653 -> 15000: error forwarding port 15000 to pod 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c, uid : port forward into network namespace "/var/run/netns/94f3dc9b-e633-4bdc-8ecb-b2e0fcb5f6fe": failed to connect to localhost:15000 inside namespace 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.959109Z error klog an error occurred forwarding 46331 -> 15020: error forwarding port 15020 to pod 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26, uid : port forward into network namespace "/var/run/netns/a590261b-e5db-442b-97a4-bcdfa2d1c415": failed to connect to localhost:15020 inside namespace 7106f11e1eb110df732fef464eb3aee49652fdd4050f8d19d1a78bbbcc7f5b26: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.959134Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:33313/debug/ndsz": EOF 2025-08-04T15:00:08.959196Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.960736Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:46331/debug/ndsz": EOF 2025-08-04T15:00:08.960792Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.960841Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:38523/config_dump?include_eds=true": EOF 2025-08-04T15:00:08.960850Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:42653/config_dump?include_eds=true": EOF 2025-08-04T15:00:08.960936Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.960957Z error port forward failed: lost connection to pod 2025-08-04T15:00:08.962406Z error klog an error occurred forwarding 36497 -> 15020: error forwarding port 15020 to pod 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c, uid : port forward into network namespace "/var/run/netns/94f3dc9b-e633-4bdc-8ecb-b2e0fcb5f6fe": failed to connect to localhost:15020 inside namespace 9cb1ddf6cccd7132efb43b516149dde9a54eb0e80390ffe19c217fd1fbbae80c: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:08.964383Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:36497/debug/ndsz": EOF 2025-08-04T15:00:08.964422Z error port forward failed: lost connection to pod 2025-08-04T15:00:09.021432Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:38523/clusters": dial tcp 127.0.0.1:38523: connect: connection refused 2025-08-04T15:00:09.021434Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:46331/stats/prometheus": dial tcp 127.0.0.1:46331: connect: connection refused 2025-08-04T15:00:09.021502Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:36497/stats/prometheus": dial tcp 127.0.0.1:36497: connect: connection refused 2025-08-04T15:00:09.021550Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-b9mp5 for: Get "http://localhost:42653/clusters": dial tcp 127.0.0.1:42653: connect: connection refused 2025-08-04T15:00:09.021574Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:35623/clusters": dial tcp 127.0.0.1:35623: connect: connection refused 2025-08-04T15:00:09.021550Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-qwvdr for: Get "http://localhost:33313/stats/prometheus": dial tcp 127.0.0.1:33313: connect: connection refused 2025-08-04T15:00:09.021611Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-52tbs for: Get "http://localhost:33101/clusters": dial tcp 127.0.0.1:33101: connect: connection refused 2025-08-04T15:00:09.021693Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:09.036014Z error klog an error occurred forwarding 38221 -> 15020: error forwarding port 15020 to pod bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01, uid : port forward into network namespace "/var/run/netns/e9103743-e437-40ee-9b4d-7289dc89e109": failed to connect to localhost:15020 inside namespace bb48d39e65c10367e4a06a3ca3a6f1eef28e00f6ad5ce437ef4488f96839bc01: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:00:09.037811Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:38221/debug/ndsz": EOF 2025-08-04T15:00:09.037897Z error port forward failed: lost connection to pod 2025-08-04T15:00:09.038892Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-f97pt for: Get "http://localhost:38221/stats/prometheus": dial tcp 127.0.0.1:38221: connect: connection refused 2025-08-04T15:00:09.038911Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:09.047933Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:59Z/90dce98f9c-5621-4818-a316-77b33629750a 2025-08-04T15:00:09.048010Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3177s 2025-08-04T15:00:09.048785Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:00:09.194730Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:59Z/90069d9fb0-9c6f-4e2d-9dc2-438599c98119 2025-08-04T15:00:09.196356Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4660s 2025-08-04T15:00:09.197196Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:00:09.285848Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:59Z/90eb9231eb-9bc0-4e91-9353-44c588d658df 2025-08-04T15:00:09.285963Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5556s 2025-08-04T15:00:09.286640Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:00:09.490958Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:00:09.502593Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/app for: pods "statefulset-v1-0" not found 2025-08-04T15:00:09.505754Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-proxy for: pods "statefulset-v1-0" not found 2025-08-04T15:00:09.508227Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-validation for: pods "statefulset-v1-0" not found 2025-08-04T15:00:09.511705Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:00:09.511735Z info tf === SUCCEEDED: Cleanup Istio in 26.055µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:tcp-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 476006 2025-08-04T15:00:24.873107Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:24.873161Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:24.908400Z info tf === Test Framework Settings === 2025-08-04T15:00:24.908520Z info tf TestID: security_ca_custom_root RunID: c81225a6-aa94-4a81-a81e-77c9cdb1e77f NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:00:24.908526Z info tf =============================== 2025-08-04T15:00:24.908689Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a 2025-08-04T15:00:24.908772Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:00:24.908785Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:00:24.908792Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:00:24.908810Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:00:24.908818Z info tf === BEGIN: Building clusters === 2025-08-04T15:00:24.910420Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:00:24.910428Z info tf === DONE: Building clusters === 2025-08-04T15:00:24.910445Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:00:24.947227Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:00:24.947300Z info tf === SUCCESS: context setup function === 2025-08-04T15:00:24.947307Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:00:24.947322Z info tf === Istio Component Config === 2025-08-04T15:00:24.947366Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:00:24.947371Z info tf ================================ 2025-08-04T15:00:24.950469Z info tf skipping deployment as specified in the config 2025-08-04T15:00:24.950486Z info tf === SUCCEEDED: Deploy Istio in 3.173485ms [Suite=security_ca_custom_root]=== 2025-08-04T15:00:24.950494Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:00:24.957665Z info tf === SUCCEEDED: Create namespace echo1 in 7.166134ms === Generating RSA private key, 2048 bit long modulus (2 primes) ************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/ca-custom-root-1186922159/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *********************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/ca-custom-root-1186922159/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/ca-custom-root-1186922159/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:00:27.545506Z info tf === BEGIN: Create namespace external === 2025-08-04T15:00:27.558512Z info tf === SUCCEEDED: Create namespace external in 13.020651ms === 2025-08-04T15:00:27.623157Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:00:27.644947Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:00:27.644973Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/ServiceAccount-Service.345599674.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "a" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:00:27.645015Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:27.645032Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:27.645024Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:27.669776Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/istio-state-4160133607/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:00:27.677955Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:00:27.702978Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/istio-state-4160133607/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:00:27.709683Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:00:27.730185Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:27.756015Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/ServiceAccount-Service.345599674.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "a" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:00:27.756098Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/ServiceAccount-Service.345599674.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "a" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T15:00:27.756104Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/ServiceAccount-Service.345599674.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "a" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:00:27.756141Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:27.756143Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:27.756156Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:27.804171Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c81225a/_suite_context/istio-state-1263019137/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:00:27.825264Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:00:27.851228Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:00:27.862240Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:00:27.862318Z info tf === SUCCEEDED: Cleanup Istio in 68.479µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:tcp-mtls-pass-through\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 477613 2025-08-04T15:00:39.514687Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:39.514748Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:39.553554Z info tf === Test Framework Settings === 2025-08-04T15:00:39.553663Z info tf TestID: security_ca_custom_root RunID: c0b1fed0-3165-4be2-a72f-99acf05255ee NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:00:39.553669Z info tf =============================== 2025-08-04T15:00:39.553819Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed 2025-08-04T15:00:39.553904Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:00:39.553917Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:00:39.553923Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:00:39.553940Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:00:39.553949Z info tf === BEGIN: Building clusters === 2025-08-04T15:00:39.555787Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:00:39.555796Z info tf === DONE: Building clusters === 2025-08-04T15:00:39.555811Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:00:39.596975Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:00:39.597035Z info tf === SUCCESS: context setup function === 2025-08-04T15:00:39.597043Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:00:39.597054Z info tf === Istio Component Config === 2025-08-04T15:00:39.597098Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:00:39.597104Z info tf ================================ 2025-08-04T15:00:39.600252Z info tf skipping deployment as specified in the config 2025-08-04T15:00:39.600269Z info tf === SUCCEEDED: Deploy Istio in 3.222333ms [Suite=security_ca_custom_root]=== 2025-08-04T15:00:39.600276Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:00:39.672298Z info tf === SUCCEEDED: Create namespace echo1 in 72.016967ms === Generating RSA private key, 2048 bit long modulus (2 primes) *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/ca-custom-root-3385282918/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************+++++ *****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/ca-custom-root-3385282918/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ****************+++++ *****************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/ca-custom-root-3385282918/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:00:42.058368Z info tf === BEGIN: Create namespace external === 2025-08-04T15:00:42.073289Z info tf === SUCCEEDED: Create namespace external in 14.935522ms === 2025-08-04T15:00:42.137318Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:00:42.163729Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:00:42.163786Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/Service-ServiceAccount.3465549838.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:00:42.163822Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:42.163838Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:42.163852Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:42.189102Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/Service-ServiceAccount.3465549838.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:00:42.189162Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/Service-ServiceAccount.3465549838.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T15:00:42.189170Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/Service-ServiceAccount.3465549838.yaml] to ns echo1 in cluster cluster-0: apply: patch: services "client" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:00:42.189203Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:42.189231Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:42.189226Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:00:42.190997Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/istio-state-3500778606/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:00:42.211786Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:00:42.293392Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-c0b1fed/_suite_context/istio-state-3500778606/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:00:42.300079Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:00:42.300142Z info tf === SUCCEEDED: Cleanup Istio in 58.603µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 479101 2025-08-04T15:00:53.863732Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:53.863830Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:00:53.898702Z info tf === Test Framework Settings === 2025-08-04T15:00:53.898806Z info tf TestID: security_ca_custom_root RunID: 7e72df84-df89-49d5-9ab6-3f1751eaf4d9 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:00:53.898813Z info tf =============================== 2025-08-04T15:00:53.898981Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8 2025-08-04T15:00:53.899060Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:00:53.899073Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:00:53.899079Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:00:53.899095Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:00:53.899104Z info tf === BEGIN: Building clusters === 2025-08-04T15:00:53.900659Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:00:53.900667Z info tf === DONE: Building clusters === 2025-08-04T15:00:53.900684Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:00:53.939649Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:00:53.939731Z info tf === SUCCESS: context setup function === 2025-08-04T15:00:53.939739Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:00:53.939756Z info tf === Istio Component Config === 2025-08-04T15:00:53.939807Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:00:53.939814Z info tf ================================ 2025-08-04T15:00:53.943019Z info tf skipping deployment as specified in the config 2025-08-04T15:00:53.943040Z info tf === SUCCEEDED: Deploy Istio in 3.295442ms [Suite=security_ca_custom_root]=== 2025-08-04T15:00:53.943049Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:00:54.025489Z info tf === SUCCEEDED: Create namespace echo1 in 82.426038ms === Generating RSA private key, 2048 bit long modulus (2 primes) **********************************************************************************************************************************************************************************************+++++ ****************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/_suite_context/ca-custom-root-3777033761/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) **********************************************************************************************************************+++++ ******************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/_suite_context/ca-custom-root-3777033761/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ***************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/_suite_context/ca-custom-root-3777033761/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:00:55.398924Z info tf === BEGIN: Create namespace external === 2025-08-04T15:00:55.413739Z info tf === SUCCEEDED: Create namespace external in 14.825377ms === 2025-08-04T15:00:55.506592Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:00:56.016042Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.040634Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.055559Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.071656Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.071788Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.072397Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.107403Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.110217Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.110997Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.111192Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.116259Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.126145Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.126210Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.134949Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.137109Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:00:56.140845Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.156530Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.172269Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.172278Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.172681Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.208004Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.210620Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.211898Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.211933Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.226605Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.226656Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.235103Z info klog Caches are synced controller=echo 2025-08-04T15:00:56.237599Z info klog Caches are synced controller=echo 2025-08-04T15:01:16.944972Z info tf === SUCCEEDED: Deploy echo instances in 21.438356761s === 2025-08-04T15:01:16.978048Z info tf === DONE: Setup: 'security_ca_custom_root' (23.077333955s) === 2025-08-04T15:01:16.978134Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:http-plaintext[allow] (re-run 1) (0.10s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[plaintext]->server:tcp-plaintext[allow] (re-run 1) (0.12s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:http-plaintext[allow] (re-run 1) (0.09s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[plaintext]->server:tcp-plaintext[allow] (re-run 1) (0.02s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:http-mtls[allow] (re-run 1) (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/client[cluster.local]->server:tcp-mtls[allow] (re-run 1) (0.09s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:http-mtls[deny] (re-run 1) (0.08s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls[deny] (re-run 1) (0.04s) PASS tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[bar]->server:tcp-mtls-pass-through[deny] (re-run 1) (0.04s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] 2025-08-04T15:01:17.619481Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T15:01:47.619968Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000479161s)' === 2025-08-04T15:01:47.620108Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T15:01:47.620108Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T15:01:47.620113Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T15:01:47.667760Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state3213586914/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:01:47.677401Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:01:47.703418Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state3213586914/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:01:47.708399Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:01:48.045835Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/10342cd4f2c-e470-46e3-961b-2b00f974e049 2025-08-04T15:01:48.046243Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4253s 2025-08-04T15:01:48.046904Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:01:48.138081Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/10390168d8b-3ee9-4064-ae63-a00825a908fe 2025-08-04T15:01:48.138194Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5173s 2025-08-04T15:01:48.138967Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:01:48.158683Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:01:48.241445Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d046f891-25b6-4817-ba9d-54f934666915 2025-08-04T15:01:48.241572Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6206s 2025-08-04T15:01:48.242398Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:01:48.459369Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:01:48.567350Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (31.01s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 1) (31.01s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] 2025-08-04T15:01:48.625453Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T15:02:18.626655Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.001197224s)' === 2025-08-04T15:02:18.626760Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T15:02:18.626764Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T15:02:18.626785Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T15:02:18.652368Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3966746662/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:18.672392Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:18.692122Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3966746662/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:18.696247Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:18.883209Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d24bb023-6612-444c-ab55-cd50c7436632 2025-08-04T15:02:18.883417Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2563s 2025-08-04T15:02:18.884234Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:18.995753Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d2362de2-5a95-4e88-b517-3cc0a05fc2f7 2025-08-04T15:02:18.996101Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3689s 2025-08-04T15:02:18.997063Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:19.165860Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103adb93b4c-44ea-48b5-84b9-33f1609d7469 2025-08-04T15:02:19.166061Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5389s 2025-08-04T15:02:19.167062Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:19.181726Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:19.486100Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:02:19.506063Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (30.95s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (re-run 1) (30.95s) === RUN TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] 2025-08-04T15:02:19.577793Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.0.207:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.0.207:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T15:02:49.579338Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.001537153s)' === 2025-08-04T15:02:49.579493Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T15:02:49.579508Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T15:02:49.579519Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T15:02:49.627975Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state4067292846/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:49.651356Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:49.706342Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state4067292846/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:49.711531Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:49.839618Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103bf16e402-1ea1-42b1-be26-4d56927f6323 2025-08-04T15:02:49.839837Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2599s 2025-08-04T15:02:49.840679Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:49.945709Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/1030509eff1-7a28-42c2-a7b3-d6f3eb04a914 2025-08-04T15:02:49.946042Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3661s 2025-08-04T15:02:49.946597Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:50.038106Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103feb08170-7fe9-42e7-b3cf-7f9270a68030 2025-08-04T15:02:50.038349Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4584s 2025-08-04T15:02:50.039028Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:50.309687Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found --- FAIL: TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.06s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 1) (31.06s) === RUN TestTrustDomainValidation/From_primary-0 2025-08-04T15:01:17.002886Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T15:02:50.635853Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m33.632959824s)' === 2025-08-04T15:02:50.635958Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T15:02:50.635974Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T15:02:50.635963Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T15:02:50.696876Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/_test_context/istio-state629703718/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:50.797657Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:51.170811Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:51.706581Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/_test_context/istio-state629703718/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:51.714484Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:51.858869Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/1035d765583-3341-45a4-976c-e79fe0e7b6b0 2025-08-04T15:02:51.859373Z info tf istioctl ([x internal-debug --all configz]): completed after 1.2232s 2025-08-04T15:02:51.859840Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:51.942430Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103232b3395-a6e9-434a-a039-ea2b29f9ee21 2025-08-04T15:02:51.942510Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.3063s 2025-08-04T15:02:51.943318Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.035371Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103c4323535-6d2f-4e25-9e3d-fc2b35f9bdeb 2025-08-04T15:02:52.035523Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.3993s 2025-08-04T15:02:52.036450Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.094833Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:52.136269Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:52.266133Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:02:52.358586Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainValidation/From_primary-0 (95.42s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation/From_primary-0 (re-run 1) (95.42s) === RUN TestTrustDomainValidation 2025-08-04T15:01:16.978809Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T15:02:52.422804Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m35.443981067s)' === 2025-08-04T15:02:52.422944Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T15:02:52.422961Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T15:02:52.422945Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T15:02:52.499960Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/_test_context/istio-state2135028956/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:52.525184Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:52.561898Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/_test_context/istio-state2135028956/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:52.578151Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:52.641622Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/10366412d13-3288-4ef7-a216-02fd78dc9a40 2025-08-04T15:02:52.642159Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2189s 2025-08-04T15:02:52.642710Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.725531Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103a4cd1f51-8c28-4228-98ca-1bc3d4d97ea7 2025-08-04T15:02:52.725643Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3024s 2025-08-04T15:02:52.726443Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.824602Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d0bdbed0-e693-451a-a128-bd6da3247907 2025-08-04T15:02:52.824712Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4015s 2025-08-04T15:02:52.825431Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.873199Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:53.174171Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:53.260617Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found --- FAIL: TestTrustDomainValidation (96.44s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainValidation (re-run 1) (96.44s) 2025-08-04T15:02:53.415475Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-08-04T15:02:53.415509Z info tf === Suite "security_ca_custom_root" run time: 1m59.514825871s === 2025-08-04T15:02:53.415528Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-08-04T15:02:53.416577Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:02:53.416596Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:02:53.416629Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:02:53.462893Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/_suite_context/istio-state-529206437/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:53.469358Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:53.491319Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/_suite_context/istio-state-529206437/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:53.498550Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:53.889436Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:02:53Z/1040dcdd209-6505-4a2f-9b03-d34588d92b7d 2025-08-04T15:02:53.889759Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4728s 2025-08-04T15:02:53.890305Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:53.983247Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:02:53Z/104a901f102-31df-4972-be84-af83a13cc057 2025-08-04T15:02:53.983481Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5665s 2025-08-04T15:02:53.984171Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:54.013092Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:54.075760Z warn tf cluster/pod cluster-0/echo1/c-v1-7bdf498cbb-qlf47 found warming resources (dynamic_warming_clusters) on attempt 1 2025-08-04T15:02:54.130489Z warn tf cluster/pod cluster-0/echo1/statefulset-v1-0 found warming resources (dynamic_warming_clusters) on attempt 1 2025-08-04T15:02:54.134347Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:02:53Z/104290aece9-feb7-4f83-b479-dab0d1a1829a 2025-08-04T15:02:54.134548Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7175s 2025-08-04T15:02:54.135326Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:54.272084Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:02:54.320863Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:55.642619Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:02:55.642712Z info tf === SUCCEEDED: Cleanup Istio in 84.109µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root DONE 2 runs, 40 tests, 17 failures in 468.382s exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0/server:server-naked-foo\[success\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 480711 2025-08-04T15:03:07.275884Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:03:07.275934Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:03:07.302957Z info tf === Test Framework Settings === 2025-08-04T15:03:07.303086Z info tf TestID: security_ca_custom_root RunID: add91499-0a23-4299-b9d2-7166c0c29b86 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:03:07.303095Z info tf =============================== 2025-08-04T15:03:07.303228Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149 2025-08-04T15:03:07.303311Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:03:07.303323Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:03:07.303330Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:03:07.303343Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:03:07.303352Z info tf === BEGIN: Building clusters === 2025-08-04T15:03:07.306119Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:03:07.306453Z info tf === DONE: Building clusters === 2025-08-04T15:03:07.307140Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:03:07.341505Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:03:07.341552Z info tf === SUCCESS: context setup function === 2025-08-04T15:03:07.341563Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:03:07.341595Z info tf === Istio Component Config === 2025-08-04T15:03:07.341675Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:03:07.341683Z info tf ================================ 2025-08-04T15:03:07.346615Z info tf skipping deployment as specified in the config 2025-08-04T15:03:07.346644Z info tf === SUCCEEDED: Deploy Istio in 5.069063ms [Suite=security_ca_custom_root]=== 2025-08-04T15:03:07.346659Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:03:07.423571Z info tf === SUCCEEDED: Create namespace echo1 in 76.89363ms === Generating RSA private key, 2048 bit long modulus (2 primes) *************************************+++++ ****************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/ca-custom-root-2860327158/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ********************+++++ **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/ca-custom-root-2860327158/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/ca-custom-root-2860327158/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:03:08.874492Z info tf === BEGIN: Create namespace external === 2025-08-04T15:03:08.880931Z info tf === SUCCEEDED: Create namespace external in 6.448642ms === 2025-08-04T15:03:08.964718Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:03:08.979804Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:03:08.979842Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/Service.3309648120.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:03:08.979871Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:03:08.979873Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:03:08.979894Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:03:09.041593Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/istio-state-2889781411/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:03:09.060771Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:03:09.075743Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/istio-state-2889781411/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:03:09.079872Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:03:09.098440Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:03:09.630327Z error klog an error occurred forwarding 33749 -> 15000: error forwarding port 15000 to pod 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3, uid : port forward into network namespace "/var/run/netns/93822058-f54b-4d2f-9a0d-21e2a9f0121d": failed to connect to localhost:15000 inside namespace 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.630331Z error klog an error occurred forwarding 37677 -> 15000: error forwarding port 15000 to pod f580f31d512be155be18990ec45bd8fface1aaf5009e6afcc00d01cc6230aa6d, uid : port forward into network namespace "/var/run/netns/9ae4dde2-0e2c-45c5-b2bc-05792570ba86": failed to connect to localhost:15000 inside namespace f580f31d512be155be18990ec45bd8fface1aaf5009e6afcc00d01cc6230aa6d: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.631238Z error klog an error occurred forwarding 43537 -> 15020: error forwarding port 15020 to pod f580f31d512be155be18990ec45bd8fface1aaf5009e6afcc00d01cc6230aa6d, uid : port forward into network namespace "/var/run/netns/9ae4dde2-0e2c-45c5-b2bc-05792570ba86": failed to connect to localhost:15020 inside namespace f580f31d512be155be18990ec45bd8fface1aaf5009e6afcc00d01cc6230aa6d: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.631423Z error klog an error occurred forwarding 45597 -> 15000: error forwarding port 15000 to pod f0ee4f5de6f9d9f2ec2cf25edda9c655e2c64fb351356a9311dceb516b41ac46, uid : port forward into network namespace "/var/run/netns/f406fb5f-edf0-499b-a54e-acddffe1fa3e": failed to connect to localhost:15000 inside namespace f0ee4f5de6f9d9f2ec2cf25edda9c655e2c64fb351356a9311dceb516b41ac46: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.631605Z error klog an error occurred forwarding 46557 -> 15020: error forwarding port 15020 to pod a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e, uid : port forward into network namespace "/var/run/netns/d973829f-bd48-4c89-862c-8661e71c50c7": failed to connect to localhost:15020 inside namespace a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.632164Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.632213Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.632166Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:33749/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.632177Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:37677/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.632942Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43537/debug/ndsz": EOF 2025-08-04T15:03:09.632989Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.633133Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:46557/debug/ndsz": EOF 2025-08-04T15:03:09.633147Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-77zbd for: Get "http://localhost:45597/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.633205Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:33749/clusters": dial tcp 127.0.0.1:33749: connect: connection refused 2025-08-04T15:03:09.633232Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.633229Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:37677/clusters": dial tcp 127.0.0.1:37677: connect: connection refused 2025-08-04T15:03:09.633301Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.633308Z error klog an error occurred forwarding 43981 -> 15020: error forwarding port 15020 to pod ff2ee3c630c3360fcfdc4ae5f93a1338a17872c1a8dcac9cf67476b789c289c8, uid : port forward into network namespace "/var/run/netns/f985578b-e774-4746-8eb7-bae321bbf340": failed to connect to localhost:15020 inside namespace ff2ee3c630c3360fcfdc4ae5f93a1338a17872c1a8dcac9cf67476b789c289c8: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.633675Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:43537/stats/prometheus": dial tcp 127.0.0.1:43537: connect: connection refused 2025-08-04T15:03:09.633672Z error klog an error occurred forwarding 45333 -> 15020: error forwarding port 15020 to pod f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1, uid : port forward into network namespace "/var/run/netns/4d550f3b-6619-4b88-bdcf-4581e0d5db25": failed to connect to localhost:15020 inside namespace f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.633702Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-77zbd for: Get "http://localhost:45597/clusters": dial tcp 127.0.0.1:45597: connect: connection refused 2025-08-04T15:03:09.633784Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:46557/stats/prometheus": dial tcp 127.0.0.1:46557: connect: connection refused 2025-08-04T15:03:09.634930Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-qlf47 for: Get "http://localhost:43981/debug/ndsz": EOF 2025-08-04T15:03:09.635106Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.635179Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:45333/debug/ndsz": EOF 2025-08-04T15:03:09.635279Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.635423Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-qlf47 for: Get "http://localhost:43981/stats/prometheus": dial tcp 127.0.0.1:43981: connect: connection refused 2025-08-04T15:03:09.635825Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:45333/stats/prometheus": dial tcp 127.0.0.1:45333: connect: connection refused 2025-08-04T15:03:09.762253Z error klog an error occurred forwarding 46631 -> 15000: error forwarding port 15000 to pod ff2ee3c630c3360fcfdc4ae5f93a1338a17872c1a8dcac9cf67476b789c289c8, uid : port forward into network namespace "/var/run/netns/f985578b-e774-4746-8eb7-bae321bbf340": failed to connect to localhost:15000 inside namespace ff2ee3c630c3360fcfdc4ae5f93a1338a17872c1a8dcac9cf67476b789c289c8: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.764001Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-qlf47 for: Get "http://localhost:46631/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.764014Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.764886Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-qlf47 for: Get "http://localhost:46631/clusters": dial tcp 127.0.0.1:46631: connect: connection refused 2025-08-04T15:03:09.774863Z error klog an error occurred forwarding 40449 -> 15000: error forwarding port 15000 to pod f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1, uid : port forward into network namespace "/var/run/netns/4d550f3b-6619-4b88-bdcf-4581e0d5db25": failed to connect to localhost:15000 inside namespace f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.777101Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:40449/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.777339Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.777949Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:40449/clusters": dial tcp 127.0.0.1:40449: connect: connection refused 2025-08-04T15:03:09.778758Z error klog an error occurred forwarding 45911 -> 15000: error forwarding port 15000 to pod 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb, uid : port forward into network namespace "/var/run/netns/c437e33d-098e-401a-8642-a0eb0f34746b": failed to connect to localhost:15000 inside namespace 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.778802Z error klog an error occurred forwarding 44419 -> 15020: error forwarding port 15020 to pod 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb, uid : port forward into network namespace "/var/run/netns/c437e33d-098e-401a-8642-a0eb0f34746b": failed to connect to localhost:15020 inside namespace 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.780256Z error klog an error occurred forwarding 37995 -> 15020: error forwarding port 15020 to pod c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6, uid : port forward into network namespace "/var/run/netns/deff1afe-2457-4346-bac4-b066a04fec20": failed to connect to localhost:15020 inside namespace c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.780455Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:45911/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.780528Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.780649Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.780723Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:44419/debug/ndsz": EOF 2025-08-04T15:03:09.780755Z error klog an error occurred forwarding 36429 -> 15020: error forwarding port 15020 to pod 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3, uid : port forward into network namespace "/var/run/netns/93822058-f54b-4d2f-9a0d-21e2a9f0121d": failed to connect to localhost:15020 inside namespace 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.781061Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:45911/clusters": dial tcp 127.0.0.1:45911: connect: connection refused 2025-08-04T15:03:09.781449Z error klog an error occurred forwarding 46613 -> 15020: error forwarding port 15020 to pod f0ee4f5de6f9d9f2ec2cf25edda9c655e2c64fb351356a9311dceb516b41ac46, uid : port forward into network namespace "/var/run/netns/f406fb5f-edf0-499b-a54e-acddffe1fa3e": failed to connect to localhost:15020 inside namespace f0ee4f5de6f9d9f2ec2cf25edda9c655e2c64fb351356a9311dceb516b41ac46: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.781518Z error klog an error occurred forwarding 43187 -> 15000: error forwarding port 15000 to pod a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e, uid : port forward into network namespace "/var/run/netns/d973829f-bd48-4c89-862c-8661e71c50c7": failed to connect to localhost:15000 inside namespace a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.781744Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:44419/stats/prometheus": dial tcp 127.0.0.1:44419: connect: connection refused 2025-08-04T15:03:09.781813Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:37995/debug/ndsz": EOF 2025-08-04T15:03:09.781954Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.782430Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:36429/debug/ndsz": EOF 2025-08-04T15:03:09.782463Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.782530Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:37995/stats/prometheus": dial tcp 127.0.0.1:37995: connect: connection refused 2025-08-04T15:03:09.782894Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-77zbd for: Get "http://localhost:46613/debug/ndsz": EOF 2025-08-04T15:03:09.783031Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.783182Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:43187/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.783208Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:36429/stats/prometheus": dial tcp 127.0.0.1:36429: connect: connection refused 2025-08-04T15:03:09.783266Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.783378Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-77zbd for: Get "http://localhost:46613/stats/prometheus": dial tcp 127.0.0.1:46613: connect: connection refused 2025-08-04T15:03:09.783704Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:43187/clusters": dial tcp 127.0.0.1:43187: connect: connection refused 2025-08-04T15:03:09.797482Z error klog an error occurred forwarding 42399 -> 15000: error forwarding port 15000 to pod c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6, uid : port forward into network namespace "/var/run/netns/deff1afe-2457-4346-bac4-b066a04fec20": failed to connect to localhost:15000 inside namespace c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:09.799438Z error port forward failed: lost connection to pod 2025-08-04T15:03:09.799442Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:42399/config_dump?include_eds=true": EOF 2025-08-04T15:03:09.800823Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:42399/clusters": dial tcp 127.0.0.1:42399: connect: connection refused 2025-08-04T15:03:09.801095Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-45s5k_echo1") 2025-08-04T15:03:09.803845Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-t9bdn_echo1") 2025-08-04T15:03:09.861518Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:03:09.900559Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:03Z/1106ae575b6-3ee9-48bb-a269-f2982e4cde50 2025-08-04T15:03:09.900790Z info tf istioctl ([x internal-debug --all configz]): completed after 0.9207s 2025-08-04T15:03:09.901612Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:03:09.922162Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-t9bdn_echo1") 2025-08-04T15:03:09.923946Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-45s5k_echo1") 2025-08-04T15:03:10.003754Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-t9bdn_echo1") 2025-08-04T15:03:10.004938Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-45s5k_echo1") 2025-08-04T15:03:10.014082Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:03Z/110be8df6e8-28df-47bd-b611-97b3b807c3d0 2025-08-04T15:03:10.014174Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.0341s 2025-08-04T15:03:10.014975Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:03:10.038748Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:03:10.121209Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-t9bdn_echo1") 2025-08-04T15:03:10.152154Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-bar-v1-dbb69ff9f-45s5k_echo1") 2025-08-04T15:03:10.169388Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:03Z/1100ae72d5a-11da-44ee-bff7-39f20ea70492 2025-08-04T15:03:10.169501Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.1894s 2025-08-04T15:03:10.170325Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:03:10.228669Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("naked-v1-57d66b659c-t9bdn_echo1") 2025-08-04T15:03:10.239277Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:03:10.243749Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/Service.3309648120.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:03:10.243779Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/Service.3309648120.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-08-04T15:03:10.243794Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/Service.3309648120.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:03:10.243849Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:03:10.243852Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:03:10.243868Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:03:10.346669Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: failed retrieving: pods "external-v1-56749b6f5c-pnsdj" not found in the "external" namespace) 2025-08-04T15:03:10.380841Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/istio-state-1853956594/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:03:10.421003Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:03:10.474527Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-add9149/_suite_context/istio-state-1853956594/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:03:10.480822Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:03:10.574370Z error klog an error occurred forwarding 37245 -> 15020: error forwarding port 15020 to pod 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb, uid : port forward into network namespace "/var/run/netns/c437e33d-098e-401a-8642-a0eb0f34746b": failed to connect to localhost:15020 inside namespace 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.577582Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.577602Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:37245/debug/ndsz": EOF 2025-08-04T15:03:10.579185Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:37245/stats/prometheus": dial tcp 127.0.0.1:37245: connect: connection refused 2025-08-04T15:03:10.606084Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-c56579cdb-77zbd_echo1") 2025-08-04T15:03:10.607027Z error klog an error occurred forwarding 39623 -> 15020: error forwarding port 15020 to pod ff2ee3c630c3360fcfdc4ae5f93a1338a17872c1a8dcac9cf67476b789c289c8, uid : sandbox ff2ee3c630c3360fcfdc4ae5f93a1338a17872c1a8dcac9cf67476b789c289c8 is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.608438Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-qlf47 for: Get "http://localhost:39623/debug/ndsz": EOF 2025-08-04T15:03:10.608501Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.608965Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-qlf47 for: Get "http://localhost:39623/stats/prometheus": dial tcp 127.0.0.1:39623: connect: connection refused 2025-08-04T15:03:10.627068Z error klog an error occurred forwarding 42751 -> 15000: error forwarding port 15000 to pod f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1, uid : port forward into network namespace "/var/run/netns/4d550f3b-6619-4b88-bdcf-4581e0d5db25": failed to connect to localhost:15000 inside namespace f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.628719Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:42751/config_dump?include_eds=true": EOF 2025-08-04T15:03:10.628739Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.629447Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:42751/clusters": dial tcp 127.0.0.1:42751: connect: connection refused 2025-08-04T15:03:10.653243Z error klog an error occurred forwarding 39113 -> 15000: error forwarding port 15000 to pod c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6, uid : port forward into network namespace "/var/run/netns/deff1afe-2457-4346-bac4-b066a04fec20": failed to connect to localhost:15000 inside namespace c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.654689Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:39113/config_dump?include_eds=true": EOF 2025-08-04T15:03:10.654811Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.654900Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:03:10.655291Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:39113/clusters": dial tcp 127.0.0.1:39113: connect: connection refused 2025-08-04T15:03:10.669258Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-08-04T15:03:10.674257Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo1") 2025-08-04T15:03:10.674494Z error klog an error occurred forwarding 33021 -> 15020: error forwarding port 15020 to pod 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3, uid : sandbox 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3 is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.676237Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:33021/debug/ndsz": EOF 2025-08-04T15:03:10.676316Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.676831Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:33021/stats/prometheus": dial tcp 127.0.0.1:33021: connect: connection refused 2025-08-04T15:03:10.677172Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("c-v1-7bdf498cbb-qlf47_echo1") 2025-08-04T15:03:10.678890Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("headless-v1-c56579cdb-77zbd_echo1") 2025-08-04T15:03:10.682222Z error klog an error occurred forwarding 45179 -> 15000: error forwarding port 15000 to pod 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb, uid : port forward into network namespace "/var/run/netns/c437e33d-098e-401a-8642-a0eb0f34746b": failed to connect to localhost:15000 inside namespace 27d47cb8b4396deea24408aba702d0827812e7a7abd4001b29e9c52f2713c7cb: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.682216Z error klog an error occurred forwarding 35399 -> 15020: error forwarding port 15020 to pod c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6, uid : port forward into network namespace "/var/run/netns/deff1afe-2457-4346-bac4-b066a04fec20": failed to connect to localhost:15020 inside namespace c76242286f4cacb97da0940de658d079bd69920616e219f0f32cb3178f75fed6: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.682404Z error klog an error occurred forwarding 44173 -> 15000: error forwarding port 15000 to pod 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3, uid : sandbox 11dd3aa65688862c2f2f7e5c69c1b1650ef5a5e4307ad87bbbcf7160fe3440d3 is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.682501Z error klog an error occurred forwarding 33813 -> 15020: error forwarding port 15020 to pod a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e, uid : port forward into network namespace "/var/run/netns/d973829f-bd48-4c89-862c-8661e71c50c7": failed to connect to localhost:15020 inside namespace a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.682688Z error klog an error occurred forwarding 42451 -> 15020: error forwarding port 15020 to pod f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1, uid : sandbox f9364403d7b879a7b19ef280c5d756e326e085000c4dadb34a5f1ee8b95ecef1 is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.683673Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:35399/debug/ndsz": EOF 2025-08-04T15:03:10.683731Z error klog an error occurred forwarding 37169 -> 15000: error forwarding port 15000 to pod a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e, uid : port forward into network namespace "/var/run/netns/d973829f-bd48-4c89-862c-8661e71c50c7": failed to connect to localhost:15000 inside namespace a7328df7c21a6056f1e3c4115d5173cadc0badca105b88ea40c1c630485d606e: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:03:10.683743Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.683921Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.683933Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:45179/config_dump?include_eds=true": EOF 2025-08-04T15:03:10.683970Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.683988Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:33813/debug/ndsz": EOF 2025-08-04T15:03:10.684015Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:44173/config_dump?include_eds=true": EOF 2025-08-04T15:03:10.684132Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.684294Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:42451/debug/ndsz": EOF 2025-08-04T15:03:10.684408Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.684463Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-9wx2j for: Get "http://localhost:35399/stats/prometheus": dial tcp 127.0.0.1:35399: connect: connection refused 2025-08-04T15:03:10.684805Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:33813/stats/prometheus": dial tcp 127.0.0.1:33813: connect: connection refused 2025-08-04T15:03:10.684824Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-cp6bc for: Get "http://localhost:45179/clusters": dial tcp 127.0.0.1:45179: connect: connection refused 2025-08-04T15:03:10.684890Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-r4vst for: Get "http://localhost:44173/clusters": dial tcp 127.0.0.1:44173: connect: connection refused 2025-08-04T15:03:10.684998Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-mtbzf for: Get "http://localhost:42451/stats/prometheus": dial tcp 127.0.0.1:42451: connect: connection refused 2025-08-04T15:03:10.685280Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:37169/config_dump?include_eds=true": EOF 2025-08-04T15:03:10.685294Z error port forward failed: lost connection to pod 2025-08-04T15:03:10.685824Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-ntpcf for: Get "http://localhost:37169/clusters": dial tcp 127.0.0.1:37169: connect: connection refused 2025-08-04T15:03:10.707370Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:03Z/110c1058370-3b23-4745-b10c-9692b427da45 2025-08-04T15:03:10.707689Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4635s 2025-08-04T15:03:10.708472Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:03:10.738736Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:03:10.767324Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:03:10.941797Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:03Z/110bfafcecd-03d9-4f24-9dae-728aae8254cc 2025-08-04T15:03:10.941920Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6977s 2025-08-04T15:03:10.942751Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:03:11.098769Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:03Z/110cbad90ee-382a-4d0d-874d-4aff7d1066ef 2025-08-04T15:03:11.098860Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.8547s 2025-08-04T15:03:11.099521Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:03:11.101076Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/headless-v1-c56579cdb-77zbd/istio-validation for: pods "headless-v1-c56579cdb-77zbd" not found 2025-08-04T15:03:11.102851Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/naked-v1-57d66b659c-t9bdn/app for: pods "naked-v1-57d66b659c-t9bdn" not found 2025-08-04T15:03:11.104622Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-6f49f7bdcb-cpqmf/istio-proxy for: pods "proxyless-grpc-v1-6f49f7bdcb-cpqmf" not found 2025-08-04T15:03:11.106724Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/proxyless-grpc-v1-6f49f7bdcb-cpqmf/app for: pods "proxyless-grpc-v1-6f49f7bdcb-cpqmf" not found 2025-08-04T15:03:11.109028Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-bar-v1-dbb69ff9f-45s5k/app for: pods "server-naked-bar-v1-dbb69ff9f-45s5k" not found 2025-08-04T15:03:11.110825Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-alt-v1-645b4f7b87-jcfkm/app for: pods "server-naked-foo-alt-v1-645b4f7b87-jcfkm" not found 2025-08-04T15:03:11.112720Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/server-naked-foo-v1-f6f5f8cb7-sfslr/app for: pods "server-naked-foo-v1-f6f5f8cb7-sfslr" not found 2025-08-04T15:03:11.277626Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:03:11.501456Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/app for: pods "statefulset-v1-0" not found 2025-08-04T15:03:11.503557Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-proxy for: pods "statefulset-v1-0" not found 2025-08-04T15:03:11.505176Z warn tf Unable to get logs for cluster/pod/container: cluster-0/echo1/statefulset-v1-0/istio-validation for: pods "statefulset-v1-0" not found 2025-08-04T15:03:11.508021Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:03:11.508043Z info tf === SUCCEEDED: Cleanup Istio in 17.254µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainAliasSecureNaming$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 482064 2025-08-04T15:03:22.464663Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:03:22.464736Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:03:22.495189Z info tf === Test Framework Settings === 2025-08-04T15:03:22.495277Z info tf TestID: security_ca_custom_root RunID: 9317357c-2337-49ad-8c54-86c367c90cc6 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:03:22.495286Z info tf =============================== 2025-08-04T15:03:22.495465Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357 2025-08-04T15:03:22.495539Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:03:22.495551Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:03:22.495557Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:03:22.495573Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:03:22.495581Z info tf === BEGIN: Building clusters === 2025-08-04T15:03:22.497088Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:03:22.497095Z info tf === DONE: Building clusters === 2025-08-04T15:03:22.497111Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:03:22.548936Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:03:22.549140Z info tf === SUCCESS: context setup function === 2025-08-04T15:03:22.549150Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:03:22.549167Z info tf === Istio Component Config === 2025-08-04T15:03:22.549231Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:03:22.549239Z info tf ================================ 2025-08-04T15:03:22.552818Z info tf skipping deployment as specified in the config 2025-08-04T15:03:22.552840Z info tf === SUCCEEDED: Deploy Istio in 3.684353ms [Suite=security_ca_custom_root]=== 2025-08-04T15:03:22.552849Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:03:22.624181Z info tf === SUCCEEDED: Create namespace echo1 in 71.305042ms === Generating RSA private key, 2048 bit long modulus (2 primes) *************************************+++++ *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/_suite_context/ca-custom-root-2062261046/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *****************************************************************************************************+++++ ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/_suite_context/ca-custom-root-2062261046/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ******************************************************************************************************************************************************************+++++ *************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/_suite_context/ca-custom-root-2062261046/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:03:24.055427Z info tf === BEGIN: Create namespace external === 2025-08-04T15:03:24.067226Z info tf === SUCCEEDED: Create namespace external in 11.81104ms === 2025-08-04T15:03:24.136563Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:03:24.714111Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.738187Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.749914Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.752877Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.756134Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.759248Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.773371Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.775180Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.778011Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.779931Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.804957Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.806589Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.810131Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.814954Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.818456Z info klog Waiting for caches to sync controller=echo 2025-08-04T15:03:24.839329Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.850386Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.853144Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.856614Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.860089Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.873543Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.876054Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.878480Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.880866Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.905390Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.906867Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.911295Z info klog Caches are synced controller=echo 2025-08-04T15:03:24.919050Z info klog Caches are synced controller=echo 2025-08-04T15:03:47.220666Z info tf === SUCCEEDED: Deploy echo instances in 23.084086973s === 2025-08-04T15:03:47.249389Z info tf === DONE: Setup: 'security_ca_custom_root' (24.752265689s) === 2025-08-04T15:03:47.249423Z info tf === BEGIN: Test Run: 'security_ca_custom_root' === === RUN TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] 2025-08-04T15:03:47.265883Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.0.225:50670->172.30.81.109:443: read: connection reset by peer 2025-08-04T15:04:17.271955Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006066436s)' === 2025-08-04T15:04:17.272042Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T15:04:17.272044Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T15:04:17.272042Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T15:04:17.320196Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state4043568911/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:17.340003Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:17.371191Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state4043568911/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:17.376867Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:17.543169Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120870d1f9c-a650-43b0-a959-06f09b46ce80 2025-08-04T15:04:17.543307Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2709s 2025-08-04T15:04:17.544019Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:17.650980Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/1208bde2a95-a37e-4e6d-a4a0-39e88e6da2a5 2025-08-04T15:04:17.651191Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3788s 2025-08-04T15:04:17.651981Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:17.797635Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:17.815471Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120b465925e-5500-47dd-9c88-8f338c23fb6f 2025-08-04T15:04:17.815779Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5433s 2025-08-04T15:04:17.816292Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.140802Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:18.191092Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (30.96s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 2) (30.96s) PASS tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-bar[fail] (re-run 2) (0.04s) === RUN TestTrustDomainAliasSecureNaming/From_primary-0 2025-08-04T15:03:47.265657Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T15:04:18.260677Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (30.995017536s)' === 2025-08-04T15:04:18.260751Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T15:04:18.260775Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T15:04:18.260789Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T15:04:18.323276Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1759726586/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:18.328301Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:18.350468Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1759726586/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:18.355717Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:18.576404Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120331dbab6-11e7-472c-87bd-326da21d0660 2025-08-04T15:04:18.576751Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3156s 2025-08-04T15:04:18.577350Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.693706Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/12034b6d49e-8e43-4e48-9fc0-f4f447f37e56 2025-08-04T15:04:18.693796Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4327s 2025-08-04T15:04:18.694494Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.836659Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/1204d5419cb-01dc-49d2-9943-0ff2f4bc19ee 2025-08-04T15:04:18.836815Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5757s 2025-08-04T15:04:18.838016Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.839239Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:19.120081Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:19.153698Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming/From_primary-0 (31.90s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 2) (31.90s) === RUN TestTrustDomainAliasSecureNaming 2025-08-04T15:03:47.250005Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T15:04:19.162951Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (31.912940646s)' === 2025-08-04T15:04:19.163054Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T15:04:19.163067Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T15:04:19.163092Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T15:04:19.229485Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/_test_context/istio-state4271446251/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:19.252252Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:19.314017Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/_test_context/istio-state4271446251/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:19.337491Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:19.475941Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120caed5cd7-b938-4842-833f-f264071ccab2 2025-08-04T15:04:19.476380Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3131s 2025-08-04T15:04:19.477040Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:19.625951Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/1209ceefdb5-d45a-47dd-b8c7-b697df09214a 2025-08-04T15:04:19.626105Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4629s 2025-08-04T15:04:19.626757Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:19.741809Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120fd302c8f-2d8c-4ffc-be5a-b81a490a338c 2025-08-04T15:04:19.742038Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5788s 2025-08-04T15:04:19.742910Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:19.754787Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory --- FAIL: TestTrustDomainAliasSecureNaming (32.86s) FAIL tests/integration/security/ca_custom_root.TestTrustDomainAliasSecureNaming (re-run 2) (32.86s) 2025-08-04T15:04:20.105189Z info tf === FAILED: Test Run: 'security_ca_custom_root' (exitCode: 1) === 2025-08-04T15:04:20.105264Z info tf === Suite "security_ca_custom_root" run time: 57.608149926s === 2025-08-04T15:04:20.105289Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/trace.yaml 2025-08-04T15:04:20.105636Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:20.105677Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:20.105671Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:20.162868Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:20.192787Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/_suite_context/istio-state-3061058423/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:20.200847Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:20.306117Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:20.306514Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/_suite_context/istio-state-3061058423/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:20.393825Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:20.694740Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:20Z/12133c2d7f3-35e7-4fdd-884d-dea642bb5063 2025-08-04T15:04:20.694842Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5888s 2025-08-04T15:04:20.695543Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:20.869030Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:20Z/121aad603b7-e271-4c1e-9e98-757625caa265 2025-08-04T15:04:20.869197Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7631s 2025-08-04T15:04:20.869956Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:20.973585Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:20Z/1215df4149d-a85b-40a5-9cff-fa24d6ebb09f 2025-08-04T15:04:20.973974Z info tf istioctl ([x internal-debug --all configz]): completed after 0.8679s 2025-08-04T15:04:20.974509Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:21.337520Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:21.424096Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:21.785025Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:21.979359Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:04:21.979436Z info tf === SUCCEEDED: Cleanup Istio in 72.194µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:http-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 483455 2025-08-04T15:04:33.212457Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:04:33.212523Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:04:33.245381Z info tf === Test Framework Settings === 2025-08-04T15:04:33.245468Z info tf TestID: security_ca_custom_root RunID: 3da115cf-e1a1-4846-8a14-71fae86b2fd4 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:04:33.245474Z info tf =============================== 2025-08-04T15:04:33.245632Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c 2025-08-04T15:04:33.245702Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:04:33.245710Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:04:33.245715Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:04:33.245729Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:04:33.245736Z info tf === BEGIN: Building clusters === 2025-08-04T15:04:33.247044Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:04:33.247052Z info tf === DONE: Building clusters === 2025-08-04T15:04:33.247064Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:04:33.283538Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:04:33.283574Z info tf === SUCCESS: context setup function === 2025-08-04T15:04:33.283581Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:04:33.283591Z info tf === Istio Component Config === 2025-08-04T15:04:33.283629Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:04:33.283634Z info tf ================================ 2025-08-04T15:04:33.286064Z info tf skipping deployment as specified in the config 2025-08-04T15:04:33.286078Z info tf === SUCCEEDED: Deploy Istio in 2.49372ms [Suite=security_ca_custom_root]=== 2025-08-04T15:04:33.286084Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:04:33.362570Z info tf === SUCCEEDED: Create namespace echo1 in 76.477441ms === Generating RSA private key, 2048 bit long modulus (2 primes) ****************************************************************************************************************************************************************************************************************************************************************************************************+++++ ****************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/ca-custom-root-1472643054/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ****************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/ca-custom-root-1472643054/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *********************************************************************************************************************************+++++ *********+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/ca-custom-root-1472643054/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:04:34.639322Z info tf === BEGIN: Create namespace external === 2025-08-04T15:04:34.646840Z info tf === SUCCEEDED: Create namespace external in 7.535414ms === 2025-08-04T15:04:34.707653Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:04:34.726477Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:04:34.726513Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/Service.1240740758.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:04:34.726542Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:34.726575Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:34.726561Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:34.821899Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/istio-state-3048282005/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:34.830486Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:34.872158Z error klog an error occurred forwarding 42543 -> 15000: error forwarding port 15000 to pod f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292, uid : port forward into network namespace "/var/run/netns/fe4748c5-155d-4f47-b9bc-edf9fdba4d03": failed to connect to localhost:15000 inside namespace f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:34.874250Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:42543/config_dump?include_eds=true": EOF 2025-08-04T15:04:34.874320Z error port forward failed: lost connection to pod 2025-08-04T15:04:34.874895Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:42543/clusters": dial tcp 127.0.0.1:42543: connect: connection refused 2025-08-04T15:04:34.912111Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:34.917803Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:35.002116Z error klog an error occurred forwarding 45741 -> 15000: error forwarding port 15000 to pod 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360, uid : port forward into network namespace "/var/run/netns/f76e4324-2b20-4f4a-9ac2-db724078e489": failed to connect to localhost:15000 inside namespace 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.002181Z error klog an error occurred forwarding 38979 -> 15020: error forwarding port 15020 to pod 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff, uid : port forward into network namespace "/var/run/netns/86ff27d1-e31d-485d-b137-c77a61fb8bf4": failed to connect to localhost:15020 inside namespace 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.001985Z error klog an error occurred forwarding 42925 -> 15000: error forwarding port 15000 to pod 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff, uid : port forward into network namespace "/var/run/netns/86ff27d1-e31d-485d-b137-c77a61fb8bf4": failed to connect to localhost:15000 inside namespace 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.002078Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/istio-state-3048282005/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:35.002222Z error klog an error occurred forwarding 38487 -> 15020: error forwarding port 15020 to pod 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360, uid : port forward into network namespace "/var/run/netns/f76e4324-2b20-4f4a-9ac2-db724078e489": failed to connect to localhost:15020 inside namespace 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.005642Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:38979/debug/ndsz": EOF 2025-08-04T15:04:35.005715Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:45741/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.005810Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.005876Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.005900Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:42925/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.005909Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.005909Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:38487/debug/ndsz": EOF 2025-08-04T15:04:35.006000Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.006411Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:38487/stats/prometheus": dial tcp 127.0.0.1:38487: connect: connection refused 2025-08-04T15:04:35.006430Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:42925/clusters": dial tcp 127.0.0.1:42925: connect: connection refused 2025-08-04T15:04:35.006453Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:45741/clusters": dial tcp 127.0.0.1:45741: connect: connection refused 2025-08-04T15:04:35.006516Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:38979/stats/prometheus": dial tcp 127.0.0.1:38979: connect: connection refused 2025-08-04T15:04:35.009472Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:35.033567Z error klog an error occurred forwarding 45317 -> 15000: error forwarding port 15000 to pod 2ac95ccaa8ddf487e5602957d9c7f3aa133bcc704c32b4643444f9e55f4b384b, uid : sandbox 2ac95ccaa8ddf487e5602957d9c7f3aa133bcc704c32b4643444f9e55f4b384b is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.033581Z error klog an error occurred forwarding 41305 -> 15000: error forwarding port 15000 to pod 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1, uid : port forward into network namespace "/var/run/netns/80c64b2a-d629-4fa3-ad3a-11cd7c3c18fb": failed to connect to localhost:15000 inside namespace 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.035475Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qxc42 for: Get "http://localhost:45317/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.035517Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.035577Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:41305/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.035680Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.036349Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:41305/clusters": dial tcp 127.0.0.1:41305: connect: connection refused 2025-08-04T15:04:35.036371Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qxc42 for: Get "http://localhost:45317/clusters": dial tcp 127.0.0.1:45317: connect: connection refused 2025-08-04T15:04:35.040662Z error klog an error occurred forwarding 46837 -> 15020: error forwarding port 15020 to pod 2ac95ccaa8ddf487e5602957d9c7f3aa133bcc704c32b4643444f9e55f4b384b, uid : sandbox 2ac95ccaa8ddf487e5602957d9c7f3aa133bcc704c32b4643444f9e55f4b384b is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.040853Z error klog an error occurred forwarding 33825 -> 15020: error forwarding port 15020 to pod 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3, uid : port forward into network namespace "/var/run/netns/7e982577-5f64-4d22-ba9f-d2a40252e9d9": failed to connect to localhost:15020 inside namespace 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.041188Z error klog an error occurred forwarding 44765 -> 15020: error forwarding port 15020 to pod f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292, uid : port forward into network namespace "/var/run/netns/fe4748c5-155d-4f47-b9bc-edf9fdba4d03": failed to connect to localhost:15020 inside namespace f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.041210Z error klog an error occurred forwarding 37591 -> 15000: error forwarding port 15000 to pod fda5190e427ac0152aa32719e91e7a42f01ad5680f726283fac0b89284989194, uid : port forward into network namespace "/var/run/netns/c2c7b291-58d1-4201-a718-735ee3e0325a": failed to connect to localhost:15000 inside namespace fda5190e427ac0152aa32719e91e7a42f01ad5680f726283fac0b89284989194: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.042284Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qxc42 for: Get "http://localhost:46837/debug/ndsz": EOF 2025-08-04T15:04:35.042298Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.042326Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:33825/debug/ndsz": EOF 2025-08-04T15:04:35.042377Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.042657Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-5cbv4 for: Get "http://localhost:37591/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.042678Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.042674Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:44765/debug/ndsz": EOF 2025-08-04T15:04:35.042687Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.042919Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/a-v1-58d94949bd-qxc42 for: Get "http://localhost:46837/stats/prometheus": dial tcp 127.0.0.1:46837: connect: connection refused 2025-08-04T15:04:35.042925Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:33825/stats/prometheus": dial tcp 127.0.0.1:33825: connect: connection refused 2025-08-04T15:04:35.043306Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-5cbv4 for: Get "http://localhost:37591/clusters": dial tcp 127.0.0.1:37591: connect: connection refused 2025-08-04T15:04:35.043310Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:44765/stats/prometheus": dial tcp 127.0.0.1:44765: connect: connection refused 2025-08-04T15:04:35.043501Z error klog an error occurred forwarding 40341 -> 15020: error forwarding port 15020 to pod 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1, uid : port forward into network namespace "/var/run/netns/80c64b2a-d629-4fa3-ad3a-11cd7c3c18fb": failed to connect to localhost:15020 inside namespace 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.043810Z error klog an error occurred forwarding 38657 -> 15020: error forwarding port 15020 to pod 0f99da81e1a2816e3b56d32842620cd97d1b80964aec39d8df9d55100ad2691a, uid : port forward into network namespace "/var/run/netns/ae9a580e-6a1d-4cef-8b2c-76fb2d88de57": failed to connect to localhost:15020 inside namespace 0f99da81e1a2816e3b56d32842620cd97d1b80964aec39d8df9d55100ad2691a: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.045771Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:40341/debug/ndsz": EOF 2025-08-04T15:04:35.045779Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:38657/debug/ndsz": EOF 2025-08-04T15:04:35.045832Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.045867Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.046199Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:38657/stats/prometheus": dial tcp 127.0.0.1:38657: connect: connection refused 2025-08-04T15:04:35.046339Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:40341/stats/prometheus": dial tcp 127.0.0.1:40341: connect: connection refused 2025-08-04T15:04:35.063026Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:30Z/1262acfee07-a2d1-479d-8817-34ee1f085438 2025-08-04T15:04:35.063094Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3362s 2025-08-04T15:04:35.063609Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:35.075767Z error klog an error occurred forwarding 38017 -> 15020: error forwarding port 15020 to pod fda5190e427ac0152aa32719e91e7a42f01ad5680f726283fac0b89284989194, uid : port forward into network namespace "/var/run/netns/c2c7b291-58d1-4201-a718-735ee3e0325a": failed to connect to localhost:15020 inside namespace fda5190e427ac0152aa32719e91e7a42f01ad5680f726283fac0b89284989194: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.077633Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.077681Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-5cbv4 for: Get "http://localhost:38017/debug/ndsz": EOF 2025-08-04T15:04:35.078561Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/b-v1-754696cfd9-5cbv4 for: Get "http://localhost:38017/stats/prometheus": dial tcp 127.0.0.1:38017: connect: connection refused 2025-08-04T15:04:35.082029Z error klog an error occurred forwarding 37313 -> 15000: error forwarding port 15000 to pod 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3, uid : port forward into network namespace "/var/run/netns/7e982577-5f64-4d22-ba9f-d2a40252e9d9": failed to connect to localhost:15000 inside namespace 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.083843Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.083861Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:37313/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.084579Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:37313/clusters": dial tcp 127.0.0.1:37313: connect: connection refused 2025-08-04T15:04:35.103088Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:35.103238Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:35.194108Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:35.234186Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:35.252744Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:30Z/126c6b1d871-a694-4cf0-b766-9217c5ff01f7 2025-08-04T15:04:35.253003Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5261s 2025-08-04T15:04:35.253602Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:35.296644Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:35.346582Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:35.441916Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:35.448774Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:30Z/126b0037238-fde3-4b5d-908a-7d818e0d6087 2025-08-04T15:04:35.448863Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7219s 2025-08-04T15:04:35.449782Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:35.516896Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:35.527023Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:04:35.747115Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/Service.1240740758.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:04:35.747190Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/Service.1240740758.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated) === 2025-08-04T15:04:35.747200Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/Service.1240740758.yaml] to ns external in cluster cluster-0: apply: patch: services "external" is forbidden: unable to create new content in namespace external because it is being terminated 2025-08-04T15:04:35.747237Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:35.747291Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:35.747273Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:35.825184Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/istio-state-1181087199/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:35.842223Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:35.881070Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:35.899282Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-3da115c/_suite_context/istio-state-1181087199/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:35.914287Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:35.957174Z error klog an error occurred forwarding 36519 -> 15020: error forwarding port 15020 to pod 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1, uid : port forward into network namespace "/var/run/netns/80c64b2a-d629-4fa3-ad3a-11cd7c3c18fb": failed to connect to localhost:15020 inside namespace 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.957676Z error klog an error occurred forwarding 43135 -> 15000: error forwarding port 15000 to pod 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1, uid : port forward into network namespace "/var/run/netns/80c64b2a-d629-4fa3-ad3a-11cd7c3c18fb": failed to connect to localhost:15000 inside namespace 48c7e2548adea91af22b6bf31eb4551bc7317ff8c6e38efb660d4c816ba782d1: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.958089Z error klog an error occurred forwarding 43181 -> 15020: error forwarding port 15020 to pod 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff, uid : port forward into network namespace "/var/run/netns/86ff27d1-e31d-485d-b137-c77a61fb8bf4": failed to connect to localhost:15020 inside namespace 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.958745Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:36519/debug/ndsz": EOF 2025-08-04T15:04:35.958778Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.959189Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:43135/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.959202Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.959406Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:36519/stats/prometheus": dial tcp 127.0.0.1:36519: connect: connection refused 2025-08-04T15:04:35.959686Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:43181/debug/ndsz": EOF 2025-08-04T15:04:35.959691Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.959939Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/sotw-v1-867646b8b5-s5tvg for: Get "http://localhost:43135/clusters": dial tcp 127.0.0.1:43135: connect: connection refused 2025-08-04T15:04:35.960322Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:43181/stats/prometheus": dial tcp 127.0.0.1:43181: connect: connection refused 2025-08-04T15:04:35.960782Z error klog an error occurred forwarding 40711 -> 15020: error forwarding port 15020 to pod 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3, uid : sandbox 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3 is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.961306Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:35.962465Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:40711/debug/ndsz": EOF 2025-08-04T15:04:35.962525Z error klog an error occurred forwarding 43419 -> 15020: error forwarding port 15020 to pod f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292, uid : port forward into network namespace "/var/run/netns/fe4748c5-155d-4f47-b9bc-edf9fdba4d03": failed to connect to localhost:15020 inside namespace f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.962553Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.962566Z error klog an error occurred forwarding 46847 -> 15000: error forwarding port 15000 to pod 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff, uid : port forward into network namespace "/var/run/netns/86ff27d1-e31d-485d-b137-c77a61fb8bf4": failed to connect to localhost:15000 inside namespace 4b2c1a43171c5b5c51f0b7f837deb1d13c15be6eed0103e17afdfd4a51c5b5ff: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.962829Z error klog an error occurred forwarding 34935 -> 15020: error forwarding port 15020 to pod 0f99da81e1a2816e3b56d32842620cd97d1b80964aec39d8df9d55100ad2691a, uid : port forward into network namespace "/var/run/netns/ae9a580e-6a1d-4cef-8b2c-76fb2d88de57": failed to connect to localhost:15020 inside namespace 0f99da81e1a2816e3b56d32842620cd97d1b80964aec39d8df9d55100ad2691a: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.963019Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:40711/stats/prometheus": dial tcp 127.0.0.1:40711: connect: connection refused 2025-08-04T15:04:35.964621Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:46847/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.964665Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:43419/debug/ndsz": EOF 2025-08-04T15:04:35.964736Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.964773Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.964810Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:34935/debug/ndsz": EOF 2025-08-04T15:04:35.964813Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.965129Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:43419/stats/prometheus": dial tcp 127.0.0.1:43419: connect: connection refused 2025-08-04T15:04:35.965157Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/server-v1-7cd6ccbd65-lp6tb for: Get "http://localhost:46847/clusters": dial tcp 127.0.0.1:46847: connect: connection refused 2025-08-04T15:04:35.965565Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:34935/stats/prometheus": dial tcp 127.0.0.1:34935: connect: connection refused 2025-08-04T15:04:35.967452Z error klog an error occurred forwarding 35091 -> 15000: error forwarding port 15000 to pod f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292, uid : port forward into network namespace "/var/run/netns/fe4748c5-155d-4f47-b9bc-edf9fdba4d03": failed to connect to localhost:15000 inside namespace f38bcd64b9be8c0ef9bd429f9e3f3c9348f07d7527130f00a090b61eefade292: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.968250Z error klog an error occurred forwarding 40227 -> 15020: error forwarding port 15020 to pod 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360, uid : port forward into network namespace "/var/run/netns/f76e4324-2b20-4f4a-9ac2-db724078e489": failed to connect to localhost:15020 inside namespace 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360: dial tcp [::1]:15020: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.969166Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:35091/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.969395Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.969519Z error klog an error occurred forwarding 34021 -> 15000: error forwarding port 15000 to pod 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3, uid : sandbox 53bbd86957f826490f97125121f9e0c3cc4e31807a33d47e1e4775adc61126e3 is not running: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.969590Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/client-v1-b45b6bf85-82tdk for: Get "http://localhost:35091/clusters": dial tcp 127.0.0.1:35091: connect: connection refused 2025-08-04T15:04:35.969865Z error tf Unable to get execute command "debug/ndsz" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:40227/debug/ndsz": EOF 2025-08-04T15:04:35.969903Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.970711Z error tf Unable to get execute command "stats/prometheus" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:40227/stats/prometheus": dial tcp 127.0.0.1:40227: connect: connection refused 2025-08-04T15:04:35.970955Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:34021/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.970991Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.971438Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/c-v1-7bdf498cbb-95nd5 for: Get "http://localhost:34021/clusters": dial tcp 127.0.0.1:34021: connect: connection refused 2025-08-04T15:04:35.975792Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:35.976015Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:35.976978Z error klog an error occurred forwarding 39531 -> 15000: error forwarding port 15000 to pod 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360, uid : port forward into network namespace "/var/run/netns/f76e4324-2b20-4f4a-9ac2-db724078e489": failed to connect to localhost:15000 inside namespace 1dc1215555bac88f8a85edf9b2e681424731bad40ee060cfa56d37f8c87b2360: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:35.978652Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:39531/config_dump?include_eds=true": EOF 2025-08-04T15:04:35.978703Z error port forward failed: lost connection to pod 2025-08-04T15:04:35.979212Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/headless-v1-c56579cdb-jkn6v for: Get "http://localhost:39531/clusters": dial tcp 127.0.0.1:39531: connect: connection refused 2025-08-04T15:04:35.982658Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:30Z/126ce0bd3ec-85f9-4a83-95e9-18aaabc7c5ae 2025-08-04T15:04:35.982883Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2352s 2025-08-04T15:04:35.982964Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:35.983324Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:35.996109Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:36.010043Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.102486Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.136914Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.153185Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.153893Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:36.184817Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:36.202836Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.209634Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:30Z/126c3b93661-ad2b-4b91-a0c0-cd7e02b9c31d 2025-08-04T15:04:36.209721Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4621s 2025-08-04T15:04:36.210383Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:36.248839Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:36.252097Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.262523Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.266126Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:36.266452Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:36.270418Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.329226Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.358573Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.360125Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:36.363383Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.365180Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:30Z/126387cbd98-4874-4962-9560-6e5362f46623 2025-08-04T15:04:36.365256Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6176s 2025-08-04T15:04:36.365961Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:36.370762Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.391529Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:36.460400Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.473199Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.488074Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("proxyless-grpc-v1-6f49f7bdcb-4lhcf_echo1") 2025-08-04T15:04:36.488615Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.497209Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1") 2025-08-04T15:04:36.500801Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("server-naked-foo-alt-v1-645b4f7b87-ndxfh_echo1") 2025-08-04T15:04:36.510877Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:04:36.550886Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("a-v1-58d94949bd-qxc42_echo1") 2025-08-04T15:04:36.878567Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:37.223211Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:37.247632Z error tf Unable to get execute command "config_dump?include_eds=true" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:38655/config_dump?include_eds=true": EOF 2025-08-04T15:04:37.254392Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:37.260421Z error klog an error occurred forwarding 38655 -> 15000: error forwarding port 15000 to pod 0f99da81e1a2816e3b56d32842620cd97d1b80964aec39d8df9d55100ad2691a, uid : port forward into network namespace "/var/run/netns/ae9a580e-6a1d-4cef-8b2c-76fb2d88de57": failed to connect to localhost:15000 inside namespace 0f99da81e1a2816e3b56d32842620cd97d1b80964aec39d8df9d55100ad2691a: dial tcp [::1]:15000: connect: connection refused: Unhandled Error logger=UnhandledError 2025-08-04T15:04:37.262301Z error port forward failed: lost connection to pod 2025-08-04T15:04:37.262365Z error tf Unable to get execute command "clusters" on cluster/pod: cluster-0/echo1/statefulset-v1-0 for: Get "http://localhost:38655/clusters": EOF 2025-08-04T15:04:37.262410Z error tf dump failed: hit max attempts 5 attempts (last error: failure running port forward process: port forward: error upgrading connection: unable to upgrade connection: pod not found ("b-v1-754696cfd9-5cbv4_echo1")) 2025-08-04T15:04:37.266859Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:04:37.266893Z info tf === SUCCEEDED: Cleanup Istio in 29.403µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:tcp-mtls\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 484987 2025-08-04T15:04:47.886373Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:04:47.886421Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:04:47.917028Z info tf === Test Framework Settings === 2025-08-04T15:04:47.917097Z info tf TestID: security_ca_custom_root RunID: 396f3d8e-4dcc-4eb5-bad4-5c3e4b2e457d NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:04:47.917102Z info tf =============================== 2025-08-04T15:04:47.917262Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8 2025-08-04T15:04:47.917327Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:04:47.917336Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:04:47.917341Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:04:47.917355Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:04:47.917362Z info tf === BEGIN: Building clusters === 2025-08-04T15:04:47.918652Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:04:47.918660Z info tf === DONE: Building clusters === 2025-08-04T15:04:47.918671Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:04:47.974835Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:04:47.974900Z info tf === SUCCESS: context setup function === 2025-08-04T15:04:47.974910Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:04:47.974942Z info tf === Istio Component Config === 2025-08-04T15:04:47.975013Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:04:47.975022Z info tf ================================ 2025-08-04T15:04:47.978297Z info tf skipping deployment as specified in the config 2025-08-04T15:04:47.978319Z info tf === SUCCEEDED: Deploy Istio in 3.40396ms [Suite=security_ca_custom_root]=== 2025-08-04T15:04:47.978331Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:04:48.033116Z info tf === SUCCEEDED: Create namespace echo1 in 54.777494ms === Generating RSA private key, 2048 bit long modulus (2 primes) *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ************************************************************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/ca-custom-root-1699324504/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/ca-custom-root-1699324504/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ ***********************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/ca-custom-root-1699324504/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:04:49.263602Z info tf === BEGIN: Create namespace external === 2025-08-04T15:04:49.278241Z info tf === SUCCEEDED: Create namespace external in 14.650189ms === 2025-08-04T15:04:49.360004Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:04:49.386946Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:04:49.387065Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/ServiceAccount-Service.3547182425.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:04:49.387116Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:49.387122Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:49.387171Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:49.421543Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/istio-state-333746964/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:49.455588Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:49.485858Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:04:49.493871Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/istio-state-333746964/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:49.497768Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/ServiceAccount-Service.3547182425.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:04:49.497795Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/ServiceAccount-Service.3547182425.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T15:04:49.497801Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/ServiceAccount-Service.3547182425.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "server-naked-foo-alt" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:04:49.497836Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:49.497866Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:49.497866Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:04:49.513354Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:49.574451Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-396f3d8/_suite_context/istio-state-3401979952/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:49.581340Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:49.589363Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:04:49Z/12759b43e44-81e4-4030-a01c-7c3d6051f8af 2025-08-04T15:04:49.589435Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2019s 2025-08-04T15:04:49.590247Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:49.597582Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:04:49.605791Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:04:49.605874Z info tf === SUCCEEDED: Cleanup Istio in 76.077µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0/naked\[foo\]->server:tcp-mtls-pass-through\[allow\]$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 486413 2025-08-04T15:05:01.483249Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:05:01.483295Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:05:01.521488Z info tf === Test Framework Settings === 2025-08-04T15:05:01.521594Z info tf TestID: security_ca_custom_root RunID: 25e0fb71-a6ab-425c-8c35-0d02edfbd5f7 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:05:01.521603Z info tf =============================== 2025-08-04T15:05:01.521791Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7 2025-08-04T15:05:01.521889Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:05:01.521902Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:05:01.521911Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:05:01.521935Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:05:01.521946Z info tf === BEGIN: Building clusters === 2025-08-04T15:05:01.523621Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:05:01.523629Z info tf === DONE: Building clusters === 2025-08-04T15:05:01.523643Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:05:01.555533Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:05:01.555572Z info tf === SUCCESS: context setup function === 2025-08-04T15:05:01.555579Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:05:01.555591Z info tf === Istio Component Config === 2025-08-04T15:05:01.555647Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:05:01.555660Z info tf ================================ 2025-08-04T15:05:01.558590Z info tf skipping deployment as specified in the config 2025-08-04T15:05:01.558605Z info tf === SUCCEEDED: Deploy Istio in 3.022881ms [Suite=security_ca_custom_root]=== 2025-08-04T15:05:01.558612Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:05:01.638981Z info tf === SUCCEEDED: Create namespace echo1 in 80.36341ms === Generating RSA private key, 2048 bit long modulus (2 primes) ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++ *********************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/ca-custom-root-1273363437/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ***********************************************************************************************************************************************************************************************************************************************************************************+++++ *************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/ca-custom-root-1273363437/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ****************************************************************************************************************************************+++++ *********************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/ca-custom-root-1273363437/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:05:02.821419Z info tf === BEGIN: Create namespace external === 2025-08-04T15:05:02.838079Z info tf === SUCCEEDED: Create namespace external in 16.676403ms === 2025-08-04T15:05:02.899241Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:05:02.922441Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:05:02.922493Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/ServiceAccount-Service.2547967531.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "headless" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:05:02.922531Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:02.922540Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:02.922579Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:02.971249Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/istio-state-1111037466/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:05:02.986813Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:05:03.035590Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/istio-state-1111037466/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:05:03.038371Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:05:03.043486Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:05:03.043552Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/ServiceAccount-Service.2547967531.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "headless" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:05:03.043594Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/ServiceAccount-Service.2547967531.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "headless" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T15:05:03.043604Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/ServiceAccount-Service.2547967531.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "headless" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:05:03.043633Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:03.043667Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:03.043684Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:03.132983Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-25e0fb7/_suite_context/istio-state-3379759248/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:05:03.135664Z error tf failed to dump ndsz: hit max attempts 5 attempts (last error: failure running port forward process: building port forwarded: pod is not running. Status=Succeeded) 2025-08-04T15:05:03.142870Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:05:03.149501Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:05:03.149571Z info tf === SUCCEEDED: Cleanup Istio in 62.812µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root exec: [go test -json -test.run=^TestTrustDomainValidation$/^From_primary-0$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/ca_custom_root -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts] go test pid: 487793 2025-08-04T15:05:14.816131Z warn unable to resolve TARGET_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:05:14.816197Z warn unable to resolve LOCAL_OUT. Dir /root/afsan/istio/out/linux_ppc64le does not exist 2025-08-04T15:05:14.849785Z info tf === Test Framework Settings === 2025-08-04T15:05:14.849900Z info tf TestID: security_ca_custom_root RunID: 8adc2efb-2ad8-45da-a6c5-4b710998398b NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-08-04T15:05:14.849906Z info tf =============================== 2025-08-04T15:05:14.850065Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef 2025-08-04T15:05:14.850147Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-08-04T15:05:14.850159Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-08-04T15:05:14.850165Z info tf Environment variable KUBECONFIG unspecified, defaulting to ~/.kube/config. 2025-08-04T15:05:14.850181Z info tf Using KubeConfigs: [/root/.kube/config]. 2025-08-04T15:05:14.850188Z info tf === BEGIN: Building clusters === 2025-08-04T15:05:14.851670Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/.kube/config 2025-08-04T15:05:14.851678Z info tf === DONE: Building clusters === 2025-08-04T15:05:14.851693Z info tf === BEGIN: Setup: 'security_ca_custom_root' === 2025-08-04T15:05:14.891942Z info configmap istio-ca-root-cert is deleted 2025-08-04T15:05:14.892038Z info tf === SUCCESS: context setup function === 2025-08-04T15:05:14.892048Z info tf === BEGIN: Deploy Istio [Suite=security_ca_custom_root] === 2025-08-04T15:05:14.892076Z info tf === Istio Component Config === 2025-08-04T15:05:14.892147Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-08-04T15:05:14.892156Z info tf ================================ 2025-08-04T15:05:14.895998Z info tf skipping deployment as specified in the config 2025-08-04T15:05:14.896013Z info tf === SUCCEEDED: Deploy Istio in 3.960282ms [Suite=security_ca_custom_root]=== 2025-08-04T15:05:14.896019Z info tf === BEGIN: Create namespace echo1 === 2025-08-04T15:05:14.969765Z info tf === SUCCEEDED: Create namespace echo1 in 73.73668ms === Generating RSA private key, 2048 bit long modulus (2 primes) *************************************************************************************************************************+++++ *******************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/ca-custom-root-2514229626/workload-server-naked-foo-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) *******************************************************************************+++++ **********************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-bar-[cert|key].pem with URI SAN spiffe://trust-domain-bar/ns/echo1/sa/server-naked-bar /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/ca-custom-root-2514229626/workload-server-naked-bar-cert.pem: OK Generating RSA private key, 2048 bit long modulus (2 primes) ****************************************************************************************************************************************************************************************************************************************************************************************************+++++ *****************************************************************************************************************************************************************************************+++++ e is 65537 (0x010001) Signature ok subject= Getting CA Private Key Generated workload-server-naked-foo-alt-[cert|key].pem with URI SAN spiffe://trust-domain-foo/ns/echo1/sa/server-naked-foo-alt /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/ca-custom-root-2514229626/workload-server-naked-foo-alt-cert.pem: OK 2025-08-04T15:05:15.933417Z info tf === BEGIN: Create namespace external === 2025-08-04T15:05:15.948074Z info tf === SUCCEEDED: Create namespace external in 14.67466ms === 2025-08-04T15:05:16.028451Z info tf === BEGIN: Deploy echo instances === 2025-08-04T15:05:16.051847Z error tf === FAILED: Deploy echo instances === 2025-08-04T15:05:16.051901Z error tf failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/ServiceAccount-Service.2080655733.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "naked" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:05:16.051959Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:16.051992Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:16.052002Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:16.065099Z error tf Test setup error: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/ServiceAccount-Service.2080655733.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "naked" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:05:16.065231Z info tf === FAILED: Setup: 'security_ca_custom_root' (failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/ServiceAccount-Service.2080655733.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "naked" is forbidden: unable to create new content in namespace echo1 because it is being terminated) === 2025-08-04T15:05:16.065245Z error tf Exiting due to setup failure: failed applying YAML files [/home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/ServiceAccount-Service.2080655733.yaml] to ns echo1 in cluster cluster-0: apply: patch: serviceaccounts "naked" is forbidden: unable to create new content in namespace echo1 because it is being terminated 2025-08-04T15:05:16.065279Z error tf === Dumping Namespace external State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:16.065296Z error tf === Dumping Istio Deployment State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:16.065313Z error tf === Dumping Namespace echo1 State for [suite(security_ca_custom_root)]... 2025-08-04T15:05:16.088248Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/istio-state-26528851/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:05:16.101247Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:05:16.131834Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-8adc2ef/_suite_context/istio-state-26528851/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:05:16.137704Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:05:16.186198Z info tf === BEGIN: Cleanup Istio [Suite=security_ca_custom_root] === 2025-08-04T15:05:16.186301Z info tf === SUCCEEDED: Cleanup Istio in 94.306µs [Suite=security_ca_custom_root] === FAIL tests/integration/security/ca_custom_root === Failed === FAIL: tests/integration/security/ca_custom_root TestSecureNaming (13.07s) 2025-08-04T14:55:45.036577Z info tf === BEGIN: Test: 'security_ca_custom_root[TestSecureNaming]' === secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:226: CA certificate root-cert.pem found secure_naming_test.go:114: retry.UntilSuccessOrFail: timeout while waiting after 10 attempts (last error: CA certificate (-----BEGIN CERTIFICATE----- MIIFFDCCAvygAwIBAgIUbQ/RQ9GNnLx7rNVNUJHeizdO1TswDQYJKoZIhvcNAQEL BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjMxMjEx MTYyNDM2WhcNMzMxMjA4MTYyNDM2WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAN7V2l7b /bZsHlWY5qC1p99Au3I0VYr34i/2BIMHSnh1jHn1B6qspjHw2IkqH0SU41M3Vpr7 m06HGuuZHsq8biUfuUL6Ywt+Vyj2UnPuc6SDPRXxAkBaVGGEAs30akJPkhbbA2qv jFxfc2A8GCfnHpGYQkZOvZEQNeHPYkTDsdLEKdbdXkzJMFEGxGDgYiW7xjowDkta YlrLBIeHwRCnZ/gTjdCu6oPjCW9JsySbbfx8dYWthIDf/SGqnX8gnDn5K1W5Ip+w xaiXZ9FsEzrWd+DXx67sPAS2wl/Qe+RVFLbCGfGnfKdj0CGUtwV/hE2BLWk7fMK/ dzQR9Bc6GM06uJejZFGuVrwkJldzWW7rp6cJc5D5vJxYCRZZsuk3x6o/mg+z+meh kYeUCgsEGU1eKJB7tJ6myiYCLvgjihnnbS0fQlq+9XhHh3IlQfYvZsgPPlWZjyE6 SO0JUNgCenuvgawihEHCUvFW/VbeuhxNlcP8w03tXcZCPJd75Xe1GRboa0KEcCwx hFMWMZmqLQOZ6XZUAWZIPqKPvBCLJKCDhXwf3kZjX9Geg04+KJwsJt5wJndlWjdb LWUIoiy41Jovc3N2hME5cinTllZI4kdOrnZ+NdL5aqoppfjZDQGHESuuPY0vxJEP dcfM5Q3BiMo/K/hNCnbp1ywtvbzGqd8/b9OtAgMBAAGjQjBAMB0GA1UdDgQWBBS9 7L7n9MlfjNgR6RLw5qN+49OOYDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEANnOOVzb6CpOkkXsSwxqaFfrCwJT+L+1O H8WvVtewAmzu8TcfuVjwwSgnVMF48TbnA1KvHIWktRNGtSTyEMLN5pH4SvCyK/9w IeIOqTNyHmN/4nETT7HjbszdjfsIWYZAjcq2cyhRj+650OJdotUHbBo7iBbAi4TW IuqseH4hgrT7qDFebhcVz8ualx9UJkmR7QqWhWqOsMbcLtm4OLZCFNJseaHM/t6H seO/OH7FNqe+GMCoXX1m348iJBZ/1ZhBMReqtNMQKpGL23fQqJnUTrD3717kS+xF 36PsFP5YYtBJ9WMXfX8Wk/YQIimXpcQeuF40VAIw5EG6SxntpBjoSPET2mTj2j9X Gunoh3PjYMBOlszrQVs66Qwe+vBj6xdPMHyRI0eTiETnmVG5jUo2cJnr8ztqD6oU zCmTX2SXCE9FXJ6MifPHEXJ1OPb9IY5tcUdjxhIB2HAigFHasxBlX44Ob6nHUyJO ZqhsF8FouuzSQTRWP7soIA3HT9MlZHhRh3Uef+ZXziKZR/Uyb66uYylIWCWsAoiU Hw4wPZhG65XTVQfg7R2cz1w72v7ugXAy3xbGfgLXhFzVnjy/ttIV/TSgpdBquhTQ 1ZG9Ieiu16avJkphqkmNr/CRCdZqDGp4DGQZNo69POGphapVe7EZenpzDU787oZF tugW0JVzMZI= -----END CERTIFICATE----- ) not matching plugin cert (-----BEGIN CERTIFICATE----- MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN 8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a 3g== -----END CERTIFICATE----- )) 2025-08-04T14:55:55.043584Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestSecureNaming] (10.007001878s)' === 2025-08-04T14:55:55.043692Z error tf === Dumping Namespace external State for TestSecureNaming... 2025-08-04T14:55:55.043706Z error tf === Dumping Istio Deployment State for TestSecureNaming... 2025-08-04T14:55:55.043710Z error tf === Dumping Namespace echo1 State for TestSecureNaming... 2025-08-04T14:55:55.105596Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-state2349979250/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:55:55.111272Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:55:55.134752Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-state2349979250/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:55:55.140679Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:55:55.569888Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:55:56.850226Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:45Z/6955dcc2ba-8df9-4554-ac66-a25fecbbc93e 2025-08-04T14:55:56.850701Z info tf istioctl ([x internal-debug --all configz]): completed after 1.8067s 2025-08-04T14:55:56.851231Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:55:56.982465Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:45Z/6939fc6ddd-c95d-4204-9b25-901a73de36fb 2025-08-04T14:55:56.982636Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.9386s 2025-08-04T14:55:56.983215Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:55:57.077471Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:45Z/69f6f6080c-ae9f-4b3e-b2fa-9dc3b5339291 2025-08-04T14:55:57.077601Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.0335s 2025-08-04T14:55:57.078417Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:55:57.217242Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:55:57.531771Z error tf === Dumping Namespace istio-system State for TestSecureNaming... 2025-08-04T14:55:57.552950Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-system-state1424418493/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:55:57.559061Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:55:57.570416Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestSecureNaming/_test_context/istio-system-state1424418493/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:55:57.576526Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:55:57.633784Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:55:57.810062Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:55:57.894604Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:55:58.094703Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (31.03s) 2025-08-04T14:55:58.125325Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.0.175:41426->172.30.125.141:443: read: connection reset by peer 2025-08-04T14:56:28.126174Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.000847115s)' === 2025-08-04T14:56:28.126281Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:56:28.126282Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:56:28.126301Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:56:28.185047Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state1463783705/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:56:28.201782Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:56:28.234645Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state1463783705/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:56:28.241090Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:56:28.643055Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/7080848670-49fc-4c0e-aee4-5bfa3c21b619 2025-08-04T14:56:28.643217Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5166s 2025-08-04T14:56:28.644037Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:28.719267Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:56:28.758950Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:56:28.772230Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/7005aff96e-5f85-4be4-8c5b-51f49ea46549 2025-08-04T14:56:28.774075Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6474s 2025-08-04T14:56:28.774349Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:28.853773Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/705fff6157-8b40-429d-a097-93add59b8b31 2025-08-04T14:56:28.853922Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.7273s 2025-08-04T14:56:28.854774Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:29.031752Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0 (32.45s) 2025-08-04T14:55:58.125029Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T14:56:29.176896Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (31.051867468s)' === 2025-08-04T14:56:29.176952Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:56:29.176955Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:56:29.176982Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:56:29.211242Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1292889741/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:56:29.232081Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:56:29.565422Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1292889741/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:56:29.571094Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:56:29.636968Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:56:29.773831Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/70e8e56be2-759b-4a07-88a2-3e074eaec6bf 2025-08-04T14:56:29.773971Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5966s 2025-08-04T14:56:29.774546Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:29.910470Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/703ba519bc-11c5-4a92-8966-acb911c91377 2025-08-04T14:56:29.910941Z info tf istioctl ([x internal-debug --all configz]): completed after 0.7336s 2025-08-04T14:56:29.911455Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:30.033822Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/706f5a109a-3d2a-4630-9b12-fddc234a6222 2025-08-04T14:56:30.034011Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.8566s 2025-08-04T14:56:30.034644Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:30.043526Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:56:30.095499Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming (33.30s) 2025-08-04T14:55:58.105165Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T14:56:30.571622Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.466443941s)' === 2025-08-04T14:56:30.571755Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:56:30.571763Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:56:30.571779Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:56:30.621692Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/_test_context/istio-state3280518401/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:56:30.641316Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:56:30.671671Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainAliasSecureNaming/_test_context/istio-state3280518401/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:56:30.680399Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:56:30.866005Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/70796e6a40-27d7-45cb-aa2c-44ae0c2963f1 2025-08-04T14:56:30.866167Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2942s 2025-08-04T14:56:30.866852Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:30.982141Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/7085268827-0d82-4a7b-914c-8fd2852bdf55 2025-08-04T14:56:30.982261Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4103s 2025-08-04T14:56:30.982952Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:31.147542Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:55:58Z/70f958139d-3abb-474b-913d-b357d007148b 2025-08-04T14:56:31.148030Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5760s 2025-08-04T14:56:31.148470Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:56:31.189055Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:56:31.212089Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (30.90s) 2025-08-04T14:56:32.857563Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T14:57:02.858106Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.00053422s)' === 2025-08-04T14:57:02.858237Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T14:57:02.858237Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T14:57:02.858304Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T14:57:02.898126Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state1714524935/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:57:02.928562Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:57:02.956462Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state1714524935/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:57:02.962150Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:57:03.192825Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7173e4785f-3fb8-440a-93e1-25ac08e29063 2025-08-04T14:57:03.193024Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3343s 2025-08-04T14:57:03.194025Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:03.305823Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7173c53715-3bdd-48b2-832a-7834955d9c4c 2025-08-04T14:57:03.305954Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4473s 2025-08-04T14:57:03.307003Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:03.489969Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/711430378f-cc35-44b0-ac26-1427a63ce5da 2025-08-04T14:57:03.490530Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6318s 2025-08-04T14:57:03.490984Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:03.510725Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:57:03.533420Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (31.24s) 2025-08-04T14:57:03.762236Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === 2025-08-04T14:57:03.875905Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T14:57:33.762989Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.000747991s)' === 2025-08-04T14:57:33.763083Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T14:57:33.763113Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T14:57:33.763136Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T14:57:33.817279Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3056744855/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:57:33.849211Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:57:33.874360Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3056744855/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:57:33.884270Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:57:34.037296Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71c6b50c3c-6893-4eb6-beba-b9780a110ff2 2025-08-04T14:57:34.037445Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2739s 2025-08-04T14:57:34.038271Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:34.177546Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71b46f9d2a-9fde-4705-859d-d251cdea0e61 2025-08-04T14:57:34.177885Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4143s 2025-08-04T14:57:34.178491Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:34.281082Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:57:34.379439Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71c0f497e6-d49e-4f35-b920-4c70a07d222b 2025-08-04T14:57:34.379588Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6160s 2025-08-04T14:57:34.380398Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:57:34.583332Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:57:34.768179Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (31.05s) 2025-08-04T14:57:35.000102Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.0.178:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.0.178:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T14:58:05.000958Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.000851932s)' === 2025-08-04T14:58:05.001057Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T14:58:05.001097Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T14:58:05.001081Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T14:58:05.029849Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state39573725/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:05.047180Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:05.079898Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state39573725/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:05.088538Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:05.207281Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7139c374cb-e49c-4d35-b057-56c68c687135 2025-08-04T14:58:05.207484Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2060s 2025-08-04T14:58:05.208356Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:05.374120Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/715a9616b4-9e80-4d85-8b39-9fd9d1efe743 2025-08-04T14:58:05.375657Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3742s 2025-08-04T14:58:05.376747Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:05.475768Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:05.562646Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71f44e74d5-3187-4468-a198-b329fdc367be 2025-08-04T14:58:05.562835Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5614s 2025-08-04T14:58:05.563830Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:05.638170Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (95.91s) 2025-08-04T14:56:31.422204Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T14:58:06.048402Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m34.6261904s)' === 2025-08-04T14:58:06.048533Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T14:58:06.048544Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T14:58:06.048534Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T14:58:06.123859Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/_test_context/istio-state162675280/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:06.130455Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:06.180346Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/From_primary-0/_test_context/istio-state162675280/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:06.181111Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:06.188038Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:06.288374Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71ba8a6307-9971-441f-87f5-cdb0d055b1f6 2025-08-04T14:58:06.288818Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2401s 2025-08-04T14:58:06.289214Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:06.673344Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/711b9dcb28-03f7-4e69-9044-93518a30ceda 2025-08-04T14:58:06.673618Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6248s 2025-08-04T14:58:06.674479Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:06.767182Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:58:06.885809Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71d5ed1cd8-1e19-4114-add9-b3aca7491888 2025-08-04T14:58:06.885953Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.8372s 2025-08-04T14:58:06.886852Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (97.14s) 2025-08-04T14:56:31.404407Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T14:58:07.336102Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m35.931687503s)' === 2025-08-04T14:58:07.336202Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T14:58:07.336203Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T14:58:07.336211Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T14:58:07.385621Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/_test_context/istio-state3482874514/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:58:07.393477Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:58:07.423297Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-027f5d2/TestTrustDomainValidation/_test_context/istio-state3482874514/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:58:07.430873Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:58:07.493431Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:07.701785Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71e9f59d3d-3fb8-41f4-a970-b6b0a7a9a7ae 2025-08-04T14:58:07.702277Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3659s 2025-08-04T14:58:07.702808Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:07.836153Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/7172b42fb2-7f9d-473c-bca9-d6b6422946cc 2025-08-04T14:58:07.836383Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5000s 2025-08-04T14:58:07.837172Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:07.945930Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:58:07.996709Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:07.996802Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:58:08.002977Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:56:31Z/71c9d5aa0b-ae60-4236-977d-d236dd4f0571 2025-08-04T14:58:08.003091Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6667s 2025-08-04T14:58:08.003755Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:58:08.296255Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 1) (31.36s) 2025-08-04T14:59:16.321052Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.0.195:34416->172.30.142.25:443: read: connection reset by peer 2025-08-04T14:59:46.327289Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006227919s)' === 2025-08-04T14:59:46.327446Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:59:46.327458Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:59:46.327470Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T14:59:46.402559Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state3194475387/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:59:46.452973Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:59:46.529776Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state3194475387/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:59:46.556957Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:59:46.786132Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86806909ce-3b88-4b27-9b2b-884b9b046610 2025-08-04T14:59:46.786772Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4589s 2025-08-04T14:59:46.787080Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:46.943211Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86af36d6da-5880-40c3-bdfd-a3f9e16abb0e 2025-08-04T14:59:46.943374Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.6155s 2025-08-04T14:59:46.944256Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:47.006373Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:47.133120Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/869f0fe587-c350-4687-ad93-9a0ff682718c 2025-08-04T14:59:47.133290Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.8054s 2025-08-04T14:59:47.134305Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:47.271353Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:59:47.418331Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 1) (32.27s) 2025-08-04T14:59:16.320814Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T14:59:47.709238Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (31.388422928s)' === 2025-08-04T14:59:47.709292Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:59:47.709301Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:59:47.709297Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T14:59:47.741244Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1578390482/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:59:47.758784Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:59:47.790417Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1578390482/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:59:47.796113Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:59:47.892537Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/866b360e55-a11b-407e-a9e1-4b6db1fde31b 2025-08-04T14:59:47.892885Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1831s 2025-08-04T14:59:47.893607Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.004135Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86aad00239-e154-4b2c-b0b7-a23f0d32a528 2025-08-04T14:59:48.004289Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2945s 2025-08-04T14:59:48.005170Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.104623Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86e5bd0c12-f5ea-4922-9bf3-364fd38b4a4d 2025-08-04T14:59:48.104773Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3950s 2025-08-04T14:59:48.105433Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.142951Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:48.471786Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:59:48.474202Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming (re-run 1) (33.21s) 2025-08-04T14:59:16.304388Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T14:59:48.588530Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (32.28413147s)' === 2025-08-04T14:59:48.588652Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:59:48.588654Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:59:48.588688Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T14:59:48.634967Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/_test_context/istio-state2149072087/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T14:59:48.650092Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T14:59:48.668811Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-5f1cb77/TestTrustDomainAliasSecureNaming/_test_context/istio-state2149072087/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T14:59:48.674454Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T14:59:48.812334Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/86e021a553-d3fb-431c-aaa3-3a8a0b16d24f 2025-08-04T14:59:48.812813Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2239s 2025-08-04T14:59:48.813437Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:48.939359Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/865db2fe4b-0f2f-4506-ae1b-108209712063 2025-08-04T14:59:48.939490Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3506s 2025-08-04T14:59:48.940491Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:49.076605Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T14:59:49.076594Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T14:59:16Z/869fc9d828-2556-48cf-aaf5-50b68640c6da 2025-08-04T14:59:49.076946Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4880s 2025-08-04T14:59:49.077789Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T14:59:49.255378Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T14:59:49.411732Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow] (re-run 1) (31.01s) 2025-08-04T15:01:17.619481Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'https://server:8091': call failed from naked (cluster=cluster-0) to https://server:8091 (using https): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "https://server:8091": remote error: tls: unknown certificate authority 2025-08-04T15:01:47.619968Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]] (30.000479161s)' === 2025-08-04T15:01:47.620108Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T15:01:47.620108Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T15:01:47.620113Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]... 2025-08-04T15:01:47.667760Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state3213586914/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:01:47.677401Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:01:47.703418Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:http-mtls[allow]/_test_context/istio-state3213586914/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:01:47.708399Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:01:48.045835Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/10342cd4f2c-e470-46e3-961b-2b00f974e049 2025-08-04T15:01:48.046243Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4253s 2025-08-04T15:01:48.046904Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:01:48.138081Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/10390168d8b-3ee9-4064-ae63-a00825a908fe 2025-08-04T15:01:48.138194Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5173s 2025-08-04T15:01:48.138967Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:01:48.158683Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:01:48.241445Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d046f891-25b6-4817-ba9d-54f934666915 2025-08-04T15:01:48.241572Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.6206s 2025-08-04T15:01:48.242398Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:01:48.459369Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:01:48.567350Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow] (re-run 1) (30.95s) 2025-08-04T15:01:48.625453Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://server:8093': call failed from naked (cluster=cluster-0) to tcp://server:8093 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T15:02:18.626655Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]] (30.001197224s)' === 2025-08-04T15:02:18.626760Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T15:02:18.626764Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T15:02:18.626785Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]... 2025-08-04T15:02:18.652368Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3966746662/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:18.672392Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:18.692122Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls[allow]/_test_context/istio-state3966746662/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:18.696247Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:18.883209Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d24bb023-6612-444c-ab55-cd50c7436632 2025-08-04T15:02:18.883417Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2563s 2025-08-04T15:02:18.884234Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:18.995753Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d2362de2-5a95-4e88-b517-3cc0a05fc2f7 2025-08-04T15:02:18.996101Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3689s 2025-08-04T15:02:18.997063Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:19.165860Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103adb93b4c-44ea-48b5-84b9-33f1609d7469 2025-08-04T15:02:19.166061Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5389s 2025-08-04T15:02:19.167062Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:19.181726Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:19.486100Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:02:19.506063Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow] (re-run 1) (31.06s) 2025-08-04T15:02:19.577793Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]]' === test.go:322: failed calling naked (cluster=cluster-0)->'tcp://172.20.0.207:9000': call failed from naked (cluster=cluster-0) to tcp://172.20.0.207:9000 (using tcp): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: remote error: tls: unknown certificate authority 2025-08-04T15:02:49.579338Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]] (30.001537153s)' === 2025-08-04T15:02:49.579493Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T15:02:49.579508Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T15:02:49.579519Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]... 2025-08-04T15:02:49.627975Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state4067292846/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:49.651356Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:49.706342Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/naked[foo]->server:tcp-mtls-pass-through[allow]/_test_context/istio-state4067292846/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:49.711531Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:49.839618Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103bf16e402-1ea1-42b1-be26-4d56927f6323 2025-08-04T15:02:49.839837Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2599s 2025-08-04T15:02:49.840679Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:49.945709Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/1030509eff1-7a28-42c2-a7b3-d6f3eb04a914 2025-08-04T15:02:49.946042Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3661s 2025-08-04T15:02:49.946597Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:50.038106Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103feb08170-7fe9-42e7-b3cf-7f9270a68030 2025-08-04T15:02:50.038349Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4584s 2025-08-04T15:02:50.039028Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:50.309687Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation/From_primary-0 (re-run 1) (95.42s) 2025-08-04T15:01:17.002886Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0]' === 2025-08-04T15:02:50.635853Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation/From_primary-0] (1m33.632959824s)' === 2025-08-04T15:02:50.635958Z error tf === Dumping Namespace external State for TestTrustDomainValidation/From_primary-0... 2025-08-04T15:02:50.635974Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation/From_primary-0... 2025-08-04T15:02:50.635963Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation/From_primary-0... 2025-08-04T15:02:50.696876Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/_test_context/istio-state629703718/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:50.797657Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:51.170811Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:51.706581Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/From_primary-0/_test_context/istio-state629703718/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:51.714484Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:51.858869Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/1035d765583-3341-45a4-976c-e79fe0e7b6b0 2025-08-04T15:02:51.859373Z info tf istioctl ([x internal-debug --all configz]): completed after 1.2232s 2025-08-04T15:02:51.859840Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:51.942430Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103232b3395-a6e9-434a-a039-ea2b29f9ee21 2025-08-04T15:02:51.942510Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.3063s 2025-08-04T15:02:51.943318Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.035371Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103c4323535-6d2f-4e25-9e3d-fc2b35f9bdeb 2025-08-04T15:02:52.035523Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.3993s 2025-08-04T15:02:52.036450Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.094833Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:52.136269Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:52.266133Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:02:52.358586Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainValidation (re-run 1) (96.44s) 2025-08-04T15:01:16.978809Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainValidation]' === 2025-08-04T15:02:52.422804Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainValidation] (1m35.443981067s)' === 2025-08-04T15:02:52.422944Z error tf === Dumping Namespace external State for TestTrustDomainValidation... 2025-08-04T15:02:52.422961Z error tf === Dumping Namespace echo1 State for TestTrustDomainValidation... 2025-08-04T15:02:52.422945Z error tf === Dumping Istio Deployment State for TestTrustDomainValidation... 2025-08-04T15:02:52.499960Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/_test_context/istio-state2135028956/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:02:52.525184Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:02:52.561898Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-7e72df8/TestTrustDomainValidation/_test_context/istio-state2135028956/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:02:52.578151Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:02:52.641622Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/10366412d13-3288-4ef7-a216-02fd78dc9a40 2025-08-04T15:02:52.642159Z info tf istioctl ([x internal-debug --all configz]): completed after 0.2189s 2025-08-04T15:02:52.642710Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.725531Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103a4cd1f51-8c28-4228-98ca-1bc3d4d97ea7 2025-08-04T15:02:52.725643Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3024s 2025-08-04T15:02:52.726443Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.824602Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:01:17Z/103d0bdbed0-e693-451a-a128-bd6da3247907 2025-08-04T15:02:52.824712Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4015s 2025-08-04T15:02:52.825431Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:02:52.873199Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:53.174171Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:02:53.260617Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success] (re-run 2) (30.96s) 2025-08-04T15:03:47.265883Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]]' === test.go:322: failed calling client (cluster=cluster-0)->'http://server-naked-foo:443': call failed from client (cluster=cluster-0) to http://server-naked-foo:443 (using http): expected no error, but encountered rpc error: code = Unknown desc = 1/1 requests had errors; first error: Get "http://server-naked-foo:443": read tcp 172.20.0.225:50670->172.30.81.109:443: read: connection reset by peer 2025-08-04T15:04:17.271955Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]] (30.006066436s)' === 2025-08-04T15:04:17.272042Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T15:04:17.272044Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T15:04:17.272042Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]... 2025-08-04T15:04:17.320196Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state4043568911/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:17.340003Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:17.371191Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/server:server-naked-foo[success]/_test_context/istio-state4043568911/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:17.376867Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:17.543169Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120870d1f9c-a650-43b0-a959-06f09b46ce80 2025-08-04T15:04:17.543307Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.2709s 2025-08-04T15:04:17.544019Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:17.650980Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/1208bde2a95-a37e-4e6d-a4a0-39e88e6da2a5 2025-08-04T15:04:17.651191Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.3788s 2025-08-04T15:04:17.651981Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:17.797635Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:17.815471Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120b465925e-5500-47dd-9c88-8f338c23fb6f 2025-08-04T15:04:17.815779Z info tf istioctl ([x internal-debug --all configz]): completed after 0.5433s 2025-08-04T15:04:17.816292Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.140802Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:18.191092Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming/From_primary-0 (re-run 2) (31.90s) 2025-08-04T15:03:47.265657Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0]' === 2025-08-04T15:04:18.260677Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming/From_primary-0] (30.995017536s)' === 2025-08-04T15:04:18.260751Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T15:04:18.260775Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T15:04:18.260789Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming/From_primary-0... 2025-08-04T15:04:18.323276Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1759726586/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:18.328301Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:18.350468Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/From_primary-0/_test_context/istio-state1759726586/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:18.355717Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:18.576404Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120331dbab6-11e7-472c-87bd-326da21d0660 2025-08-04T15:04:18.576751Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3156s 2025-08-04T15:04:18.577350Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.693706Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/12034b6d49e-8e43-4e48-9fc0-f4f447f37e56 2025-08-04T15:04:18.693796Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.4327s 2025-08-04T15:04:18.694494Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.836659Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/1204d5419cb-01dc-49d2-9943-0ff2f4bc19ee 2025-08-04T15:04:18.836815Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.5757s 2025-08-04T15:04:18.838016Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:18.839239Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory 2025-08-04T15:04:19.120081Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/opentelemetry-collector-8dd5f7d78-pq9jh/opentelemetry-collector: previous terminated container "opentelemetry-collector" in pod "opentelemetry-collector-8dd5f7d78-pq9jh" not found 2025-08-04T15:04:19.153698Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk: error exec'ing into istio-system/istio-ingressgateway-5754586674-cwnmk istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory === FAIL: tests/integration/security/ca_custom_root TestTrustDomainAliasSecureNaming (re-run 2) (32.86s) 2025-08-04T15:03:47.250005Z info tf === BEGIN: Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming]' === 2025-08-04T15:04:19.162951Z info tf === DONE (failed): Test: 'security_ca_custom_root[TestTrustDomainAliasSecureNaming] (31.912940646s)' === 2025-08-04T15:04:19.163054Z error tf === Dumping Namespace external State for TestTrustDomainAliasSecureNaming... 2025-08-04T15:04:19.163067Z error tf === Dumping Istio Deployment State for TestTrustDomainAliasSecureNaming... 2025-08-04T15:04:19.163092Z error tf === Dumping Namespace echo1 State for TestTrustDomainAliasSecureNaming... 2025-08-04T15:04:19.229485Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/_test_context/istio-state4271446251/primary-0/istio-egressgateway-665464fcb8-5fbjj_istio-proxy.previous.log 2025-08-04T15:04:19.252252Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj/istio-proxy: previous terminated container "istio-proxy" in pod "istio-egressgateway-665464fcb8-5fbjj" not found 2025-08-04T15:04:19.314017Z error tf FAIL: cluster/pod/container cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy crashed/restarted 1 times. Logs: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-ca_custom_root/artifacts/security-ca-custom-root-9317357/TestTrustDomainAliasSecureNaming/_test_context/istio-state4271446251/primary-0/istio-ingressgateway-5754586674-cwnmk_istio-proxy.previous.log 2025-08-04T15:04:19.337491Z warn tf Unable to get previous logs for cluster/pod/container: cluster-0/istio-system/istio-ingressgateway-5754586674-cwnmk/istio-proxy: previous terminated container "istio-proxy" in pod "istio-ingressgateway-5754586674-cwnmk" not found 2025-08-04T15:04:19.475941Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120caed5cd7-b938-4842-833f-f264071ccab2 2025-08-04T15:04:19.476380Z info tf istioctl ([x internal-debug --all configz]): completed after 0.3131s 2025-08-04T15:04:19.477040Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:19.625951Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/1209ceefdb5-d45a-47dd-b8c7-b697df09214a 2025-08-04T15:04:19.626105Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.4629s 2025-08-04T15:04:19.626757Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:19.741809Z info adsc Received type=istio.io/debug count=1 nonce=2025-08-04T15:03:47Z/120fd302c8f-2d8c-4ffc-be5a-b81a490a338c 2025-08-04T15:04:19.742038Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5788s 2025-08-04T15:04:19.742910Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-08-04T15:04:19.754787Z warn tf Unable to get core dumps for cluster/pod: cluster-0/istio-system/istio-egressgateway-665464fcb8-5fbjj: error exec'ing into istio-system/istio-egressgateway-665464fcb8-5fbjj istio-proxy container: command terminated with exit code 255 executable file `find` not found in $PATH: No such file or directory DONE 3 runs, 44 tests, 20 failures in 608.913s exec: go version