IPv6 disconnected env - The control-plane-pki-operator pod fails to start its container because /usr/bin/control-plane-pki-operator is not found on host:


$ oc get po -n clusters-hosted-0 control-plane-pki-operator-764bdf6964-6sr79 
NAME                                          READY   STATUS                 RESTARTS   AGE
control-plane-pki-operator-764bdf6964-6sr79   0/1     CreateContainerError   0          14m


$ oc describe po -n clusters-hosted-0 control-plane-pki-operator-764bdf6964-6sr79 
Name:                 control-plane-pki-operator-764bdf6964-6sr79
Namespace:            clusters-hosted-0
Priority:             100000000
Priority Class Name:  hypershift-control-plane
Service Account:      control-plane-pki-operator
Node:                 master-0-0.ocp-edge-cluster-0.qe.lab.redhat.com/fd2e:6f44:5dd8::5c
Start Time:           Fri, 15 Dec 2023 08:28:17 -0500
Labels:               app=control-plane-pki-operator
                      hypershift.openshift.io/control-plane-component=control-plane-pki-operator
                      hypershift.openshift.io/hosted-control-plane=clusters-hosted-0
                      hypershift.openshift.io/need-management-kas-access=true
                      name=control-plane-pki-operator
                      pod-template-hash=764bdf6964
Annotations:          hypershift.openshift.io/release-image:
                        registry.ocp-edge-cluster-0.qe.lab.redhat.com:5005/ocp/4-dev-preview@sha256:a70f1994591681f58dfbdc3ea6c6699e9061b45547d7bff8baf21105d1b2e9...
                      k8s.ovn.org/pod-networks:
                        {"default":{"ip_addresses":["fd01:0:0:1::9f/64"],"mac_address":"0a:58:4b:61:6b:1d","gateway_ips":["fd01:0:0:1::1"],"routes":[{"dest":"fd01...
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ovn-kubernetes",
                            "interface": "eth0",
                            "ips": [
                                "fd01:0:0:1::9f"
                            ],
                            "mac": "0a:58:4b:61:6b:1d",
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: restricted-v2
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:               Pending
IP:                   fd01:0:0:1::9f
IPs:
  IP:           fd01:0:0:1::9f
Controlled By:  ReplicaSet/control-plane-pki-operator-764bdf6964
Containers:
  control-plane-pki-operator:
    Container ID:  
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb466525f5c382cd36bcfc4d58e5fcf9c78d0f3ffe0b8a930867529388f82dee
    Image ID:      
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /usr/bin/control-plane-pki-operator
    Args:
      operator
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     10m
      memory:  80Mi
    Environment:
      POD_NAME:                        control-plane-pki-operator-764bdf6964-6sr79 (v1:metadata.name)
      HOSTED_CONTROL_PLANE_NAMESPACE:  clusters-hosted-0 (v1:metadata.namespace)
      HOSTED_CONTROL_PLANE_NAME:       hosted-0
      CERT_ROTATION_SCALE:             24h0m0s
    Mounts:
      /etc/pki/ca-trust/extracted/pem from openshift-config-managed-trusted-ca-bundle (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6pfwh (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  openshift-config-managed-trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openshift-config-managed-trusted-ca-bundle
    Optional:  true
  kube-api-access-6pfwh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 hypershift.openshift.io/cluster=clusters-hosted-0:NoSchedule
                             hypershift.openshift.io/control-plane=true:NoSchedule
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       19m   default-scheduler  Successfully assigned clusters-hosted-0/control-plane-pki-operator-764bdf6964-6sr79 to master-0-0.ocp-edge-cluster-0.qe.lab.redhat.com
  Normal   AddedInterface  19m   multus             Add eth0 [fd01:0:0:1::9f/64] from ovn-kubernetes
  Normal   Pulling         19m   kubelet            Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb466525f5c382cd36bcfc4d58e5fcf9c78d0f3ffe0b8a930867529388f82dee"
  Normal   Pulled          18m   kubelet            Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb466525f5c382cd36bcfc4d58e5fcf9c78d0f3ffe0b8a930867529388f82dee" in 11.535551991s (11.535565677s including waiting)
  Warning  Failed          18m   kubelet            Error: container create failed: time="2023-12-15T13:28:32Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed          18m   kubelet            Error: container create failed: time="2023-12-15T13:28:33Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed          18m   kubelet            Error: container create failed: time="2023-12-15T13:28:44Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed          18m   kubelet            Error: container create failed: time="2023-12-15T13:28:56Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed          18m   kubelet            Error: container create failed: time="2023-12-15T13:29:08Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed          18m   kubelet            Error: container create failed: time="2023-12-15T13:29:20Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075868f0_7b2c_4e24_82c6_63b9138f09af.slice/crio-a01fe2261474f48722e7f9f440a2c768a9d889916ebc99ea17044e0b9834cbfc.scope/memory.events: no such file or directory"
time="2023-12-15T13:29:20Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed  17m  kubelet  Error: container create failed: time="2023-12-15T13:29:35Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075868f0_7b2c_4e24_82c6_63b9138f09af.slice/crio-8b39167ba434e26731e58e5b83c50e81f6aaeacd8f89bda323dba8e8b0704b3f.scope/memory.events: no such file or directory"
time="2023-12-15T13:29:35Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed  17m                   kubelet  Error: container create failed: time="2023-12-15T13:29:46Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed  17m                   kubelet  Error: container create failed: time="2023-12-15T13:30:00Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Warning  Failed  16m (x3 over 17m)     kubelet  (combined from similar events): Error: container create failed: time="2023-12-15T13:30:41Z" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/control-plane-pki-operator\": stat /usr/bin/control-plane-pki-operator: no such file or directory"
  Normal   Pulled  3m58s (x68 over 18m)  kubelet  Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb466525f5c382cd36bcfc4d58e5fcf9c78d0f3ffe0b8a930867529388f82dee" already present on machine

HCP version: OCP 4.15.0-ec.3
Hypershift operator image: https://quay.io/hypershift/hypershift-operator@sha256:d09c47c2eea5c95707c8c2704edda0d8fd99d3c601c810108b167f6a498c787a

100%    

    1. Install OCP 4.14 management cluster - IPv6 disconnected
    2. Install MCE 2.5 and use https://quay.io/hypershift/hypershift-operator@sha256:d09c47c2eea5c95707c8c2704edda0d8fd99d3c601c810108b167f6a498c787a as hypershift operator image
    3. Install 4.15 hosted cluster

    control-plane-pki-operator pod stuck in CreateContainerError

    control-plane-pki-operator pod is Ready