Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-76479

Openshift pods stay in a "CreateContainerError" status and show the executable file not found in $PATH" error.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.18.z
    • Node / Kubelet
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      $ oc get pods
      NAME                                                         READY   STATUS                      RESTARTS   AGE
      cluster-59fd4ffbfd-pvws6                                     1/1     Running                     0          1d
      gitops-plugin-895cc759-wth87                                 1/1     Running                     0          1d
      openshift-gitops-application-controller-0                    0/1     CrashLoopBackOff            121        1d
      openshift-gitops-applicationset-controller-8bb69d85d-xdqzs   0/1     CrashLoopBackOff            311        1d
      openshift-gitops-dex-server-5bf4bf4f8c-vd298                 1/1     Running                     0          20d
      openshift-gitops-dex-server-6876d69597-kl8xp                 0/1     Init:CreateContainerError   0          1d
      openshift-gitops-redis-7848d4b859-t8k2m                      1/1     Running                     0          20d
      openshift-gitops-repo-server-56fdcdd764-w6866                1/1     Running                     0          20d
      openshift-gitops-repo-server-6f4b8564bd-znhd5                0/1     Init:CreateContainerError   0          1d
      openshift-gitops-server-5f87669c76-cfg9w                     0/1     CrashLoopBackOff            120        1d

      $ oc get events
      LAST SEEN   TYPE      REASON    OBJECT                                                           MESSAGE
      37m         Normal    Pulled    pod/openshift-gitops-application-controller-0                    Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
      2m51s       Warning   BackOff   pod/openshift-gitops-application-controller-0                    Back-off restarting failed container argocd-application-controller in pod openshift-gitops-application-controller-0_openshift-gitops(c1a7c45c-2772-4010-a2c5-164a18dde3b1)
      42m         Normal    Pulled    pod/openshift-gitops-applicationset-controller-8bb69d85d-xdqzs   Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
      2m48s       Warning   BackOff   pod/openshift-gitops-applicationset-controller-8bb69d85d-xdqzs   Back-off restarting failed container argocd-applicationset-controller in pod openshift-gitops-applicationset-controller-8bb69d85d-xdqzs_openshift-gitops(8105b135-261e-4f08-8e01-864b0ea0ae5d)
      7m          Normal    Pulled    pod/openshift-gitops-dex-server-6876d69597-kl8xp                 Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
      2m30s       Warning   Failed    pod/openshift-gitops-dex-server-6876d69597-kl8xp                 Error: container create failed: executable file `cp` not found in $PATH: No such file or directory
      7m          Normal    Pulled    pod/openshift-gitops-repo-server-6f4b8564bd-znhd5                Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
      2m56s       Warning   Failed    pod/openshift-gitops-repo-server-6f4b8564bd-znhd5                Error: container create failed: executable file `cp` not found in $PATH: No such file or directory
      12m         Normal    Pulled    pod/openshift-gitops-server-5f87669c76-cfg9w                     Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
      2m44s       Warning   BackOff   pod/openshift-gitops-server-5f87669c76-cfg9w                     Back-off restarting failed container argocd-server in pod openshift-gitops-server-5f87669c76-cfg9w_openshift-gitops(f9a45aac-1af4-47a2-9397-24dc97364798)

      $ oc get pods
      NAME                                        READY   STATUS                 RESTARTS   AGE
      cluster-logging-operator-5bff654c87-64xbd   1/1     Running                0          2d
      instance-dqg5g                              1/1     Running                0          3d
      instance-grktk                              1/1     Running                0          2d
      instance-nkjc5                              0/1     CreateContainerError   0          2d
      instance-nsvpw                              0/1     CreateContainerError   0          2d
      instance-rnbhs                              1/1     Running                0          2d

      $ oc get events
      LAST SEEN   TYPE      REASON   OBJECT               MESSAGE
      1m37s       Normal    Pulled   pod/instance-nkjc5   Container image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:fa2cfa2ed336ce105c8dea5bfe0825407e37ef296193ae162f515213fe43c8d5" already present on machine
      56m         Warning   Failed   pod/instance-nkjc5   Error: container create failed: executable file `sh` not found in $PATH: No such file or directory
      2m23s       Normal    Pulled   pod/instance-nsvpw   Container image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:fa2cfa2ed336ce105c8dea5bfe0825407e37ef296193ae162f515213fe43c8d5" already present on machine
      17m         Warning   Failed   pod/instance-nsvpw   Error: container create failed: executable file `sh` not found in $PATH: No such file or directory

      Similar to this was also experienced in v4.6 for which bug: https://bugzilla.redhat.com/show_bug.cgi?id=1950536 and is closed.

      The temporary workaround  is to delete all the images from /var/lib/containers/storage directories and reboot which doesn't helps as a permanent solution.  

              pehunt@redhat.com Peter Hunt
              rhn-support-skohli Shubh Kohli
              None
              None
              Min Li Min Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: