Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-788

Pods are unable to start due to error "Failed to set quota limit for projid xxxxx on /var/lib/containers/storage/overlay/backingFsBlockDev: no such file or directory"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • 4.9
    • Node / CRI-O
    • None
    • Moderate
    • None
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      As described in the KCS 6844091 and already discussed in slack:
      https://coreos.slack.com/archives/CK1AE4ZCK/p1647878338011239
      https://coreos.slack.com/archives/CK1AE4ZCK/p1655367192152449

      Some customer are facing a mysterious deletion of the file /var/lib/containers/storage/overlay/backingFsBlockDev, blocking all new POD from starting.

      The issue has been identified in several cases (the list is available at the bottom of the KCS) and is impacting multiple versions (cf table below)

      The KCS 6844091 was updated to provide an easy fix: 

      sudo podman info

       and the process to add an audit rule on the node is also described in the KCS.

      Version-Release number of selected component (if applicable):

      Case number Version Provider
      03303118 4.9.35 vSphere
      03262441 4.8.x unknown
      03228366 4.8.24 vSphere
      03148155 4.8.31 vSphere
      03222783 4.9.21 unknown
      03183661 4.8.28 vSphere
      03182250 4.8.26 VSphere
      03177367 4.8.18 AWS

      How reproducible:

      It's easy to replicate the issue by removing the file */var/lib/containers/storage/overlay/backingFsBlockDev*, but this does not provide the root cause of the removal of the file in the customer clusters :) 
      

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

      Pods scheduled on the impacted node(s) are stuck in 
      

      Expected results:

      The file */var/lib/containers/storage/overlay/backingFsBlockDev* should not disappear.
      Would it be a permanent solution to include in the code the creation of the file when flagged as missing after warning about the absence of the file?
      

      Additional info:

      Related Bugzilla 2091214

              sgrunert@redhat.com Sascha Grunert
              rhn-support-vlours Vincent Lours
              Sunil Choudhary Sunil Choudhary
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: