Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-1602

4.16 nightly, CrashLoopBackOff in etcd for control-plain-machine-set-operator jobs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • None
    • False
    • None
    • False

      In 4.16.0-0.nightly-2024-04-11-095910:

       * overall-analysis-all (3 prow jobs, focusing only on this job. The other two jobs are job and job2.

      The pods.json shows:

      $ cat pods.json |jq -r '.items|.[]|select(.metadata.name == "etcd-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0")|.status.containerStatuses|.[].lastState.terminated.message'
      
      Error: open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory
      failed to create etcd client: open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory
      
      {"level":"info","ts":"2024-04-11T13:09:05.64908Z","caller":"etcdmain/grpc_proxy.go:218","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-metrics-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-metrics-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}
      
      {"level":"fatal","ts":"2024-04-11T13:09:05.649398Z","caller":"etcdmain/grpc_proxy.go:413","msg":"failed to create TLS listener","error":"open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-metrics-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.mustListenCMux\n\tgo.etcd.io/etcd/server/v3/etcdmain/grpc_proxy.go:413\ngo.etcd.io/etcd/server/v3/etcdmain.startGRPCProxy\n\tgo.etcd.io/etcd/server/v3/etcdmain/grpc_proxy.go:220\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:897\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:32\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:250"}
      
      I0411 13:09:05.935904       1 readyz.go:155] Listening on 0.0.0.0:9980
      F0411 13:09:05.936486       1 readyz.go:69] open /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-serving-ci-op-pwpw95v7-db02d-4wdsp-master-vnqzw-0.crt: no such file or directory
      

      Which could imply a secrets rotation issue.
       

            dperique@redhat.com Dennis Periquet
            dperique@redhat.com Dennis Periquet
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: