Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-20538

cadvisor/kubelet discrepancies. pod ephemeral-storage + imagefs != node.fs.usedBytes

XMLWordPrintable

    • Important
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      node.fs.usedBytes reported by /api/v1/nodes/{NODE}/proxy/stats/summary doesn't match with the results of pod ephemeral-storage usage + node.runtime.imageFs.usedBytes

      Version-Release number of selected component (if applicable):

      OCP 4.12

      How reproducible:

      All the time

      Steps to Reproduce:

      1. Get the amount of ephemeral-storage used reported by api: oc get --raw /api/v1/nodes/{NODE}/proxy/stats/summary | jq -r '.pods[] | ."ephemeral-storage".usedBytes' 
      
      2. Get the amount of usedBytes for imageFS: oc get --raw /api/v1/nodes/{NODE}/proxy/stats/summary | jq -r '"imagefs:\(.node.runtime.imageFs.usedBytes)"'
      
      3. Get the amount of usedBytes for nodeFS: oc get --raw /api/v1/nodes/{NODE}/proxy/stats/summary | jq -r '"Used:\(.node.fs.usedBytes)"'
      
      4. ephemeral-storage.usedBytes + node.runtime.imageFs.usedBytes != .node.fs.usedBytes 

      Actual results:

      - The sum of reported sizes for usedBytes imagefs + the sum of ephemeral-storage for pods on the node doesn't match to what the node.fs.usedBytes reports
      
      - Would like to understand how imagefs plays in node storage consumption.

      Expected results:

      - Reported stats should match to what is used within the node OR
      - If the calculation is not correct, please share the details of how those stats are being calculated
      - Need assistance from cAdvisor / kubelet engineer to clarify stats behavior

      Additional info:

      - Customer would like to understand:
        - How is this difference calculated?
        - What is the usage of metrics contribution to storage, node usage as customer wants to adjust some limits on ephemeral-storage
        - What is the pod ephemeral usage metrics such as imagefs , pod ephemeral usage etc and their contribution to the node disk usage?

              rh-ee-kehannon Kevin Hannon
              rhn-support-jcoscia Javier Coscia
              Sunil Choudhary Sunil Choudhary
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: