exec: [go test -json -tags=integ -timeout 180m ./tests/integration/security/cacert_rotation -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-cacert_rotation/artifacts] go test pid: 3515042 2025-12-09T11:22:12.709026Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3.x/istio-integration-test/istio-integration-3.1.4/istio/out/linux_ppc64le does not exist 2025-12-09T11:22:12.709081Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3.x/istio-integration-test/istio-integration-3.1.4/istio/out/linux_ppc64le does not exist 2025-12-09T11:22:12.746459Z info tf === Test Framework Settings === 2025-12-09T11:22:12.746559Z info tf TestID: security_cacert_rotation RunID: 8ad85c5a-730d-4ff2-a5c5-fd8fd278d250 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-cacert_rotation/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-12-09T11:22:12.746567Z info tf =============================== 2025-12-09T11:22:12.766270Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-cacert_rotation/artifacts/security-cacert-rotation-8ad85c 2025-12-09T11:22:12.766474Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-12-09T11:22:12.766492Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-12-09T11:22:12.766513Z info tf Using KubeConfigs: [/root/upi/cC/auth/kubeconfig]. 2025-12-09T11:22:12.766525Z info tf === BEGIN: Building clusters === 2025-12-09T11:22:12.770461Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cC/auth/kubeconfig 2025-12-09T11:22:12.770469Z info tf === DONE: Building clusters === 2025-12-09T11:22:12.770499Z info tf === BEGIN: Setup: 'security_cacert_rotation' === 2025-12-09T11:22:13.222108Z info configmap istio-ca-root-cert is deleted 2025-12-09T11:22:13.222160Z info tf === SUCCESS: context setup function === 2025-12-09T11:22:13.222167Z info tf === BEGIN: Deploy Istio [Suite=security_cacert_rotation] === 2025-12-09T11:22:13.222177Z info tf === Istio Component Config === 2025-12-09T11:22:13.222230Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-12-09T11:22:13.222236Z info tf ================================ 2025-12-09T11:22:13.224709Z info tf skipping deployment as specified in the config 2025-12-09T11:22:13.224729Z info tf === SUCCEEDED: Deploy Istio in 2.556454ms [Suite=security_cacert_rotation]=== 2025-12-09T11:22:13.224822Z info tf === BEGIN: Create namespace external === 2025-12-09T11:22:13.224824Z info tf === BEGIN: Create namespace echo === 2025-12-09T11:22:13.351216Z info tf === SUCCEEDED: Create namespace external in 126.407107ms === 2025-12-09T11:22:13.436645Z info tf === SUCCEEDED: Create namespace echo in 211.822038ms === 2025-12-09T11:22:15.662572Z info tf === BEGIN: Deploy echo instances === 2025-12-09T11:22:21.050310Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.098530Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.123519Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.150829Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.172306Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.183907Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.185618Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.186545Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.198820Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.198974Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.209024Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:22:21.223720Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.272760Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.284036Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.286665Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.286686Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.299172Z info klog Caches are synced controller=echo 2025-12-09T11:22:21.309873Z info klog Caches are synced controller=echo 2025-12-09T11:22:32.393727Z info tf === SUCCEEDED: Deploy echo instances in 16.731144954s === 2025-12-09T11:22:32.422572Z info tf === DONE: Setup: 'security_cacert_rotation' (19.65204843s) === 2025-12-09T11:22:32.422616Z info tf === BEGIN: Test Run: 'security_cacert_rotation' === PASS tests/integration/security/cacert_rotation.TestReachability/a/to_b (0.27s) PASS tests/integration/security/cacert_rotation.TestReachability/a (0.27s) === RUN TestReachability 2025-12-09T11:22:32.423117Z info tf === BEGIN: Test: 'security_cacert_rotation[TestReachability]' === 2025-12-09T11:22:32.481592Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0518s run.go:57: Running tests with: sources [a.echo] -> destinations [b.echo] main_test.go:114: step 1: Update CA root cert with combined root 2025-12-09T11:22:32.832785Z info configmap istio-ca-root-cert is deleted 2025-12-09T11:22:32.887807Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0549s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:34.319895Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0497s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:35.749787Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0764s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:37.402579Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.1265s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:39.986978Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0706s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:41.862288Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0886s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:43.333965Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0594s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:44.958106Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0640s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:46.413147Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0599s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:47.955368Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0689s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:49.409631Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0608s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:50.854986Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0568s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:52.364913Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0600s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:53.853216Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0536s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:55.859213Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.2221s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:23:03.219687Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 4.1794s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:23:27.014862Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.1946s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:213: failed to update pod statefulset-v1-0: etcdserver: request timed out 2025-12-09T11:24:00.808970Z info tf === DONE (failed): Test: 'security_cacert_rotation[TestReachability] (1m28.385848441s)' === 2025-12-09T11:24:00.809134Z error tf === Dumping Istio Deployment State for TestReachability... 2025-12-09T11:24:00.809138Z error tf === Dumping Namespace echo State for TestReachability... 2025-12-09T11:24:00.809189Z error tf === Dumping Namespace external State for TestReachability... 2025-12-09T11:24:01.894762Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/3870670eed-ddcf-48ad-b85a-d1cc8b39f110 2025-12-09T11:24:01.895128Z info tf istioctl ([x internal-debug --all configz]): completed after 1.0857s 2025-12-09T11:24:01.895684Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:03.070406Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/38ad7d9594-8ffc-4e26-8871-dce423c563cb 2025-12-09T11:24:03.070515Z info tf istioctl ([x internal-debug --all mcsz]): completed after 2.2611s 2025-12-09T11:24:03.071315Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:03.974912Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/383b17097e-0104-4ea7-a59c-06940a1a034d 2025-12-09T11:24:03.975003Z info tf istioctl ([x internal-debug --all clusterz]): completed after 3.1655s 2025-12-09T11:24:03.976015Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:09.942837Z error tf === Dumping Namespace istio-system State for TestReachability... --- FAIL: TestReachability (98.07s) FAIL tests/integration/security/cacert_rotation.TestReachability (98.07s) 2025-12-09T11:24:10.488795Z info tf === FAILED: Test Run: 'security_cacert_rotation' (exitCode: 1) === 2025-12-09T11:24:10.488845Z info tf === Suite "security_cacert_rotation" run time: 1m57.718361193s === 2025-12-09T11:24:10.488860Z info tf Wrote trace to /home/jenkins/workspace/sail/istio-integration-tests-suites/security-cacert_rotation/artifacts/trace.yaml 2025-12-09T11:24:10.489269Z error tf === Dumping Namespace echo State for [suite(security_cacert_rotation)]... 2025-12-09T11:24:10.489275Z error tf === Dumping Istio Deployment State for [suite(security_cacert_rotation)]... 2025-12-09T11:24:10.489290Z error tf === Dumping Namespace external State for [suite(security_cacert_rotation)]... 2025-12-09T11:24:11.130693Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/386561d45e-021d-4d77-bec1-84ba5077383a 2025-12-09T11:24:11.131135Z info tf istioctl ([x internal-debug --all configz]): completed after 0.6417s 2025-12-09T11:24:11.131906Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:11.581872Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/38cc363d91-c686-44ff-bfcc-e253e1be94f9 2025-12-09T11:24:11.582035Z info tf istioctl ([x internal-debug --all mcsz]): completed after 1.0926s 2025-12-09T11:24:11.583048Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:11.757344Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/3892a2d399-79ea-4467-9ae8-e8c4c64493aa 2025-12-09T11:24:11.757472Z info tf istioctl ([x internal-debug --all clusterz]): completed after 1.2680s 2025-12-09T11:24:11.758509Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:12.969210Z info tf === BEGIN: Cleanup Istio [Suite=security_cacert_rotation] === 2025-12-09T11:24:12.969322Z info tf === SUCCEEDED: Cleanup Istio in 105.052µs [Suite=security_cacert_rotation] === FAIL tests/integration/security/cacert_rotation DONE 3 tests, 1 failure in 120.300s exec: [go test -json -test.run=^TestReachability$ -tags=integ -timeout 180m istio.io/istio/tests/integration/security/cacert_rotation -args -istio.test.skipWorkloads=tproxy,vm -istio.test.openshift -istio.test.kube.helm.values=global.platform=openshift -istio.test.istio.enableCNI=true -istio.test.ci=true -istio.test.env=kube -istio.test.kube.deploy=false -istio.test.stableNamespaces=true -istio.test.kube.deployGatewayAPI=false -istio.test.gatewayConformance.maxTimeToConsistency=180s -istio.test.work_dir=/home/jenkins/workspace/sail/istio-integration-tests-suites/security-cacert_rotation/artifacts] go test pid: 3518199 2025-12-09T11:24:25.393074Z warn unable to resolve TARGET_OUT. Dir /root/anurag/ossm3.x/istio-integration-test/istio-integration-3.1.4/istio/out/linux_ppc64le does not exist 2025-12-09T11:24:25.393862Z warn unable to resolve LOCAL_OUT. Dir /root/anurag/ossm3.x/istio-integration-test/istio-integration-3.1.4/istio/out/linux_ppc64le does not exist 2025-12-09T11:24:25.445573Z info tf === Test Framework Settings === 2025-12-09T11:24:25.445669Z info tf TestID: security_cacert_rotation RunID: a59284c8-0c44-43b6-a55f-ba2dff890bb9 NoCleanup: false BaseDir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-cacert_rotation/artifacts Selector: FailOnDeprecation: false CIMode: true Retries: 0 StableNamespaces: true Revision: SkipWorkloads [tproxy vm] Compatibility: false Revisions: Hub: quay.io/maistra Tag: ibm-p Variant: PullPolicy: Always PullSecret: MaxDumps: 10 HelmRepo: https://istio-release.storage.googleapis.com/charts IPFamilies: [] GatewayConformanceStandardOnly: false GatewayConformanceAllowCRDsMismatch: false 2025-12-09T11:24:25.445676Z info tf =============================== 2025-12-09T11:24:25.446315Z info tf Test run dir: /home/jenkins/workspace/sail/istio-integration-tests-suites/security-cacert_rotation/artifacts/security-cacert-rotation-a59284 2025-12-09T11:24:25.446428Z info tf Test Framework Kubernetes environment Settings: Kubeconfigs: [] LoadBalancerSupported: true MCSControllerEnabled: false ControlPlaneTopology: map[] NetworkTopology: map[] ConfigTopology: map[] 2025-12-09T11:24:25.446442Z info tf Flags istio.test.kube.config and istio.test.kube.topology not specified. 2025-12-09T11:24:25.446465Z info tf Using KubeConfigs: [/root/upi/cC/auth/kubeconfig]. 2025-12-09T11:24:25.446479Z info tf === BEGIN: Building clusters === 2025-12-09T11:24:25.451993Z info tf Built Cluster: Name: cluster-0 StableName: primary-0 PrimaryCluster: cluster-0 ConfigCluster: cluster-0 Network: HTTPProxy: ProxyKubectlOnly: false Filename: /root/upi/cC/auth/kubeconfig 2025-12-09T11:24:25.452052Z info tf === DONE: Building clusters === 2025-12-09T11:24:25.452078Z info tf === BEGIN: Setup: 'security_cacert_rotation' === 2025-12-09T11:24:25.606613Z info configmap istio-ca-root-cert is deleted 2025-12-09T11:24:25.606682Z info tf === SUCCESS: context setup function === 2025-12-09T11:24:25.606694Z info tf === BEGIN: Deploy Istio [Suite=security_cacert_rotation] === 2025-12-09T11:24:25.606707Z info tf === Istio Component Config === 2025-12-09T11:24:25.606802Z info tf SystemNamespace: istio-system TelemetryNamespace: istio-system DeployIstio: false DeployEastWestGW: true Values: map[global.hub:quay.io/maistra global.imagePullPolicy:Always global.platform:openshift global.tag:ibm-p global.variant:] PrimaryClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml ConfigClusterIOPFile: tests/integration/iop-integration-test-defaults.yaml RemoteClusterIOPFile: tests/integration/iop-remote-integration-test-defaults.yaml BaseIOPFile: tests/integration/base.yaml SkipWaitForValidationWebhook: false DumpKubernetesManifests: false IstiodlessRemotes: true OperatorOptions: map[] EnableCNI: true IngressGatewayServiceName: IngressGatewayServiceNamespace: IngressGatewayIstioLabel: EgressGatewayServiceName: istio-egressgateway EgressGatewayServiceNamespace: istio-system EgressGatewayIstioLabel: egressgateway SharedMeshConfigName: ControlPlaneInstaller: 2025-12-09T11:24:25.606816Z info tf ================================ 2025-12-09T11:24:25.611517Z info tf skipping deployment as specified in the config 2025-12-09T11:24:25.611552Z info tf === SUCCEEDED: Deploy Istio in 4.849729ms [Suite=security_cacert_rotation]=== 2025-12-09T11:24:25.611660Z info tf === BEGIN: Create namespace echo === 2025-12-09T11:24:25.611656Z info tf === BEGIN: Create namespace external === 2025-12-09T11:24:25.715282Z info tf === SUCCEEDED: Create namespace echo in 103.617485ms === 2025-12-09T11:24:25.715920Z info tf === SUCCEEDED: Create namespace external in 104.279558ms === 2025-12-09T11:24:25.967408Z info tf === BEGIN: Deploy echo instances === 2025-12-09T11:24:26.509838Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.527191Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.538925Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.539446Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.539646Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.540413Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.577582Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.578227Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.589249Z info klog Waiting for caches to sync controller=echo 2025-12-09T11:24:26.610311Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.628169Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.639903Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.639899Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.639925Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.641400Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.678428Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.678493Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.694546Z info klog Caches are synced controller=echo 2025-12-09T11:24:26.712723Z error klog an error occurred forwarding 38043 -> 17070: error forwarding port 17070 to pod 488b96e06ceb092eb39c39ed91d64743b73700a202b01557b782d5135d2690a3, uid : port forward into network namespace "/var/run/netns/8f3782c6-6779-428a-89f9-cc9d3fd5da43": failed to connect to localhost:17070 inside namespace 488b96e06ceb092eb39c39ed91d64743b73700a202b01557b782d5135d2690a3: dial tcp [::1]:17070: connect: connection refused: Unhandled Error logger=UnhandledError 2025-12-09T11:24:26.714957Z error port forward failed: lost connection to pod 2025-12-09T11:24:26.742964Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-12-09T11:24:26.752030Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-x9w8q_echo") 2025-12-09T11:24:26.977428Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-x9w8q_echo") 2025-12-09T11:24:26.999250Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("statefulset-v1-0_echo") 2025-12-09T11:24:27.269790Z error port forward failed: error upgrading connection: unable to upgrade connection: pod not found ("sotw-v1-867646b8b5-x9w8q_echo") 2025-12-09T11:24:36.548532Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:24:36.573195Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:24:41.695729Z info Work item handle failed (failed connecting to grpc client to pod echo/headless-v1-c56579cdb-7wwt5 : context deadline exceeded), retry after delay 1s 2025-12-09T11:24:47.549273Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:24:47.574385Z info Work item handle failed (timeout while waiting after 8 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:24:52.696276Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:24:58.550433Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:24:58.575438Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:03.697151Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:09.552541Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:09.576402Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:14.725465Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:20.554293Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:20.578155Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:25.727434Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:31.556050Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:31.580060Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:36.728493Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:42.581411Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:42.650098Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:47.729465Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:53.583436Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:53.651243Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:25:58.730646Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:04.584235Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:04.652301Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:09.732213Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:15.585737Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:15.653879Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:20.732957Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:26.587656Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:26.655934Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:31.734952Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:37.588635Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:37.657313Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:42.736203Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:48.589546Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:48.659220Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:53.738249Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:59.590550Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:26:59.660276Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:04.740045Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:10.591837Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:10.661892Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:15.741495Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:21.593270Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:21.663931Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:26.742951Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:32.594613Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:32.665389Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:37.744540Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:43.596631Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:43.666294Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:48.746212Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:54.598014Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:54.667140Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:27:59.747711Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:05.599641Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:05.668399Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:10.748447Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:16.601231Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:16.668863Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:21.750026Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:27.602966Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:27.669527Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:32.751162Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:38.604279Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:38.670970Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:43.752443Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:49.605406Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:49.672230Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:28:54.753403Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:00.606741Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:00.673632Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:05.754628Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:11.607686Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:11.674296Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:16.756365Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:22.608969Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:22.675460Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:27.758284Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:33.610699Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:33.676253Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:38.759385Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:44.612005Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:44.676637Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:49.760318Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:55.614267Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:29:55.677890Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:00.761126Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:06.615440Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:06.679180Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:11.762296Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:17.617290Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:17.680956Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:22.763872Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:28.618428Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:28.682243Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:33.765351Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:39.620174Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:39.683843Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:44.766690Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:50.621888Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:50.685384Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:30:55.768050Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:01.623794Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:01.686636Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:06.769549Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:12.625103Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:12.687815Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:17.771132Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:23.625965Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:23.688664Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:28.771411Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:34.627214Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:34.689922Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:39.773155Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:45.628665Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:45.690237Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:50.775200Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:56.630228Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:31:56.691961Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:01.777102Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:07.631952Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:07.693649Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:12.778870Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:18.632892Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:18.695273Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:23.779936Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:29.635036Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:29.696412Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:34.781666Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:40.636690Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:40.697656Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:45.783362Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:51.637062Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:51.698992Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:32:56.785219Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:02.638536Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:02.700513Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:07.786659Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:13.640117Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:13.702085Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:18.788248Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:24.641740Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:24.703681Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:29.790300Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:35.643859Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:35.705133Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:40.791839Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:46.644120Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:46.706014Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:51.793681Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:57.644883Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:33:57.707998Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:02.795470Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:08.646042Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:08.709071Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:13.796958Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:19.647663Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:19.710860Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:24.797928Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:30.648663Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:30.711793Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:35.799942Z info Work item handle failed (timeout while waiting after 10 attempts (last error: failed starting port forwarder for pod echo/headless-v1-c56579cdb-7wwt5: failure running port forward process: building port forwarded: failed retrieving: pods "headless-v1-c56579cdb-7wwt5" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:35.800002Z error tf === FAILED: Deploy echo instances === 2025-12-09T11:34:35.800023Z error tf 9 errors occurred: * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/proxyless-grpc) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/b) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/a) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/naked) * timeout while waiting after 378 attempts (last error: no workloads ready for echo external/external) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/c) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/sotw) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/statefulset) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/headless) 2025-12-09T11:34:35.800081Z error tf === Dumping Namespace external State for [suite(security_cacert_rotation)]... 2025-12-09T11:34:35.800092Z error tf === Dumping Istio Deployment State for [suite(security_cacert_rotation)]... 2025-12-09T11:34:35.800101Z error tf === Dumping Namespace echo State for [suite(security_cacert_rotation)]... 2025-12-09T11:34:35.811298Z error tf Test setup error: 9 errors occurred: * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/proxyless-grpc) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/b) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/a) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/naked) * timeout while waiting after 378 attempts (last error: no workloads ready for echo external/external) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/c) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/sotw) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/statefulset) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/headless) 2025-12-09T11:34:35.811665Z info tf === FAILED: Setup: 'security_cacert_rotation' (9 errors occurred: * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/proxyless-grpc) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/b) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/a) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/naked) * timeout while waiting after 378 attempts (last error: no workloads ready for echo external/external) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/c) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/sotw) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/statefulset) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/headless) ) === 2025-12-09T11:34:35.814451Z error tf Exiting due to setup failure: 9 errors occurred: * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/proxyless-grpc) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/b) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/a) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/naked) * timeout while waiting after 378 attempts (last error: no workloads ready for echo external/external) * timeout while waiting after 378 attempts (last error: no workloads ready for echo echo/c) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/sotw) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/statefulset) * timeout while waiting after 58 attempts (last error: no workloads ready for echo echo/headless) 2025-12-09T11:34:35.814547Z error tf === Dumping Namespace external State for [suite(security_cacert_rotation)]... 2025-12-09T11:34:35.814545Z error tf === Dumping Istio Deployment State for [suite(security_cacert_rotation)]... 2025-12-09T11:34:35.814570Z error tf === Dumping Namespace echo State for [suite(security_cacert_rotation)]... 2025-12-09T11:34:35.968289Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:26:25Z/612b4e58db-0a6f-49fe-9e4d-51a653f54aa3 2025-12-09T11:34:35.968562Z info tf istioctl ([x internal-debug --all configz]): completed after 0.1676s 2025-12-09T11:34:35.969201Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:34:36.083149Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:26:25Z/6153f243d3-c264-4a81-a9f8-042474ddb094 2025-12-09T11:34:36.083388Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.2824s 2025-12-09T11:34:36.084330Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:34:36.176706Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:26:25Z/6197316685-00a3-428b-a984-cab5b4a07f4d 2025-12-09T11:34:36.176871Z info tf istioctl ([x internal-debug --all clusterz]): completed after 0.3759s 2025-12-09T11:34:36.177686Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:34:36.270799Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:26:25Z/614fb14886-0258-4b36-9954-2450240fd5a7 2025-12-09T11:34:36.271295Z info tf istioctl ([x internal-debug --all configz]): completed after 0.4565s 2025-12-09T11:34:36.271697Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:34:36.399405Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:26:25Z/61ef521a00-1e93-40b7-aeb1-93005e2a13fd 2025-12-09T11:34:36.399535Z info tf istioctl ([x internal-debug --all mcsz]): completed after 0.5847s 2025-12-09T11:34:36.400252Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:34:38.669646Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:26:25Z/612e00d17c-6482-48e1-bdcf-aa6b05ab8cd8 2025-12-09T11:34:38.669735Z info tf istioctl ([x internal-debug --all clusterz]): completed after 2.8549s 2025-12-09T11:34:38.670465Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:34:41.649237Z info Work item handle failed (timeout while waiting after 9 attempts (last error: failed starting port forwarder for pod echo/sotw-v1-867646b8b5-x9w8q: failure running port forward process: building port forwarded: failed retrieving: pods "sotw-v1-867646b8b5-x9w8q" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:41.713945Z info Work item handle failed (timeout while waiting after 8 attempts (last error: failed starting port forwarder for pod echo/statefulset-v1-0: failure running port forward process: building port forwarded: failed retrieving: pods "statefulset-v1-0" not found in the "echo" namespace)), retry after delay 1s 2025-12-09T11:34:41.714022Z info tf === BEGIN: Cleanup Istio [Suite=security_cacert_rotation] === 2025-12-09T11:34:41.714042Z info tf === SUCCEEDED: Cleanup Istio in 12.382µs [Suite=security_cacert_rotation] === FAIL tests/integration/security/cacert_rotation === Failed === FAIL: tests/integration/security/cacert_rotation TestReachability (98.07s) 2025-12-09T11:22:32.423117Z info tf === BEGIN: Test: 'security_cacert_rotation[TestReachability]' === 2025-12-09T11:22:32.481592Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0518s run.go:57: Running tests with: sources [a.echo] -> destinations [b.echo] main_test.go:114: step 1: Update CA root cert with combined root 2025-12-09T11:22:32.832785Z info configmap istio-ca-root-cert is deleted 2025-12-09T11:22:32.887807Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0549s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:34.319895Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0497s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:35.749787Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0764s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:37.402579Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.1265s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:39.986978Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0706s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:41.862288Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0886s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:43.333965Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0594s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:44.958106Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0640s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:46.413147Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0599s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:47.955368Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0689s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:49.409631Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0608s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:50.854986Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0568s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:52.364913Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0600s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:53.853216Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.0536s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:22:55.859213Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.2221s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:23:03.219687Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 4.1794s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:216: updating pod echo/statefulset-v1-0 on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-egressgateway-665464fcb8-skvpn on cluster cluster-0 main_test.go:216: updating pod istio-system/istio-ingressgateway-5754586674-4k9nr on cluster cluster-0 main_test.go:216: updating pod istio-system/istiod-7db5878d5f-2pmhg on cluster cluster-0 2025-12-09T11:23:27.014862Z info tf istioctl ([pc s a-v1-58d94949bd-t8c8n.echo -o json]): completed after 0.1946s main_test.go:216: updating pod echo/a-v1-58d94949bd-t8c8n on cluster cluster-0 main_test.go:216: updating pod echo/b-v1-754696cfd9-hfchp on cluster cluster-0 main_test.go:216: updating pod echo/c-v1-7bdf498cbb-7d4pk on cluster cluster-0 main_test.go:216: updating pod echo/headless-v1-c56579cdb-7wwt5 on cluster cluster-0 main_test.go:216: updating pod echo/proxyless-grpc-v1-6f49f7bdcb-dv5wb on cluster cluster-0 main_test.go:216: updating pod echo/sotw-v1-867646b8b5-x9w8q on cluster cluster-0 main_test.go:213: failed to update pod statefulset-v1-0: etcdserver: request timed out 2025-12-09T11:24:00.808970Z info tf === DONE (failed): Test: 'security_cacert_rotation[TestReachability] (1m28.385848441s)' === 2025-12-09T11:24:00.809134Z error tf === Dumping Istio Deployment State for TestReachability... 2025-12-09T11:24:00.809138Z error tf === Dumping Namespace echo State for TestReachability... 2025-12-09T11:24:00.809189Z error tf === Dumping Namespace external State for TestReachability... 2025-12-09T11:24:01.894762Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/3870670eed-ddcf-48ad-b85a-d1cc8b39f110 2025-12-09T11:24:01.895128Z info tf istioctl ([x internal-debug --all configz]): completed after 1.0857s 2025-12-09T11:24:01.895684Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:03.070406Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/38ad7d9594-8ffc-4e26-8871-dce423c563cb 2025-12-09T11:24:03.070515Z info tf istioctl ([x internal-debug --all mcsz]): completed after 2.2611s 2025-12-09T11:24:03.071315Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:03.974912Z info adsc Received type=istio.io/debug count=1 nonce=2025-12-09T11:23:38Z/383b17097e-0104-4ea7-a59c-06940a1a034d 2025-12-09T11:24:03.975003Z info tf istioctl ([x internal-debug --all clusterz]): completed after 3.1655s 2025-12-09T11:24:03.976015Z info adsc connection closed with err: rpc error: code = Unavailable desc = error reading from server: EOF 2025-12-09T11:24:09.942837Z error tf === Dumping Namespace istio-system State for TestReachability... 2025-12-09T11:22:32.531305Z info tf === BEGIN: Test: 'security_cacert_rotation[TestReachability/a]' === 2025-12-09T11:22:32.802888Z info tf === DONE (passed): Test: 'security_cacert_rotation[TestReachability/a] (271.581894ms)' === --- PASS: TestReachability/a (0.27s) 2025-12-09T11:22:32.536748Z info tf === BEGIN: Test: 'security_cacert_rotation[TestReachability/a/to_b]' === 2025-12-09T11:22:32.802793Z info tf === DONE (passed): Test: 'security_cacert_rotation[TestReachability/a/to_b] (266.03841ms)' === --- PASS: TestReachability/a/to_b (0.27s) DONE 2 runs, 3 tests, 1 failure in 749.045s exec: go version