Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16874

Error when upgrading the cluster "ImagePullBackOff: error creating read-write layer with ID * no such file or directory"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 4.12.z, 4.11.z, 4.10.z, 4.8.z
    • Node / CRI-O
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      • During the time of upgrading the cluster, Nodes failed to pull images with errors like following, which blocking the OCP upgrade:
        ~~~
        error creating read-write layer with ID "4372a0c382584d7752da058c5267d1d652d727585457a71c5ef3d4d17a951719": Stat /var/lib/containers/storage/overlay/27abd31f77c1e21b8897140edb61d1e52d48a3cd287c03796dbecb54684871d3: no such file or directory
        ~~~
      • Cu use below method to clean the nodes images storage but it is an temporary fix
        ~~~
        $ sudo -i
      1. systemctl disable kubelet
      2. systemctl disable crio
      3. reboot

      $ sudo -i

      1. rm -rf /var/lib/containers/*
      2. crio wipe -f
      3. systemctl enable --now crio
      4. systemctl enable --now kubelet
        ~~~

      The above shared steps is a temprary fix, they always need to perform the above steps before upgrading cluster and looking for permanent fix.

      Cu face this issue when upgrade from 4.8 to 4.10 and also during upgrading from 4.10 to 4.12

      Note: Openinig this bug after discussion with Peter Hunt on "https://issues.redhat.com/browse/OCPBUGS-8939"

              sgrunert@redhat.com Sascha Grunert
              rhn-support-rojain Rohan Jain (Inactive)
              None
              None
              Sunil Choudhary Sunil Choudhary
              None
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: