-
Bug
-
Resolution: Done
-
Blocker
-
None
-
None
-
False
-
None
-
False
-
-
In 4.16.0-0.nightly-2024-04-11-095910:
* overall-analysis-all (3 prow jobs, focusing only on this job. The other two jobs are job and job2.
The pods.json shows:
$ cat pods.json |jq -r '.items|.[]|select(.metadata.name == "etcd-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0")|.status.containerStatuses|.[].lastState.terminated.message' Error: open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory failed to create etcd client: open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory {"level":"info","ts":"2024-04-11T13:09:05.64908Z","caller":"etcdmain/grpc_proxy.go:218","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-metrics-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-metrics-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "} {"level":"fatal","ts":"2024-04-11T13:09:05.649398Z","caller":"etcdmain/grpc_proxy.go:413","msg":"failed to create TLS listener","error":"open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-metrics-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.mustListenCMux\n\tgo.etcd.io/etcd/server/v3/etcdmain/grpc_proxy.go:413\ngo.etcd.io/etcd/server/v3/etcdmain.startGRPCProxy\n\tgo.etcd.io/etcd/server/v3/etcdmain/grpc_proxy.go:220\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:897\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:32\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:250"} I0411 13:09:05.935904 1 readyz.go:155] Listening on 0.0.0.0:9980 F0411 13:09:05.936486 1 readyz.go:69] open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory
Which could imply a secrets rotation issue.