Uploaded image for project: 'Cloud Infrastructure Security & Compliance'
  1. Cloud Infrastructure Security & Compliance
  2. CMP-4006

High CPU and Memory Usage in File Integrity Operator (FIO 1.3.6) on OCP 4.18.z - AIDE Pods Hitting Resource Limits

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      1. Deploy OCP 4.18.22 

      2. Install File Integrity Operator 1.3.6

      3. Apply FileIntegrity object with default settings

      4. Observe resource usage: oc adm top pods -n openshift-file-integrity

      Show
      1. Deploy OCP 4.18.22  2. Install File Integrity Operator 1.3.6 3. Apply FileIntegrity object with default settings 4. Observe resource usage: oc adm top pods -n openshift-file-integrity

      Description of problem:

      OpenShift 4.18.z with File Integrity Operator 1.3.6, the aide-worker-fileintegrity DaemonSet pods show continuous CPU and memory consumption growth during hourly AIDE scan cycles. Even when resource limits are increased, memory and CPU utilization continue to rise until the new limits are almost reached.
      
      The issue was reproduced in a clean lab environment with the exact same OCP and operator versions, confirming it is not customer-specific, and not caused by configuration.

      Version-Release number of selected component (if applicable):

      I have reproduced the issue in the following environments:
      
      - OpenShift versions tested:
      
        - 4.18.22
      
        - 4.18.24
      
      - FIO client versions tested:
      
        - 1.3.6
      
        - 1.3.7

      How reproducible:

      Always

      Steps to Reproduce:

          1. Deploy OCP 4.18.22
          2. Install File Integrity Operator 1.3.6
          3. Apply FileIntegrity object with default settings
          4. Observe resource usage:
             oc adm top pods -n openshift-file-integrity

      Actual results:

      cluster(4.18.22) and operator(1.3.6) version
      ~~~
      NAME                                       CPU(cores)   MEMORY(bytes)
      aide-worker-fileintegrity-46cnt            161m         569Mi
      aide-worker-fileintegrity-svcmp            89m          355Mi
      aide-worker-fileintegrity-wcpg7            300m         286Mi
      file-integrity-operator-8479f84bd6-st4lm   2m           32Mi
      
      NAME                                       CPU(cores)   MEMORY(bytes)
      aide-worker-fileintegrity-46cnt            300m         544Mi
      aide-worker-fileintegrity-svcmp            96m          580Mi
      aide-worker-fileintegrity-wcpg7            301m         105Mi
      file-integrity-operator-8479f84bd6-st4lm   1m           31Mi
      ~~~
      
      I upgraded the File Integrity Operator to the latest version (1.3.7) on same openshift 4.18.22 but still facing same high consumption of cpu and memory.
      ~~~
      NAME                                       CPU(cores)   MEMORY(bytes)
      aide-worker-fileintegrity-742xh            299m         526Mi
      aide-worker-fileintegrity-7sqcd            296m         546Mi
      aide-worker-fileintegrity-b2bjf            239m         397Mi
      file-integrity-operator-77989587b6-zj57t   2m           26Mi
      ~~~
      
      I have reproduce the issue on 4.18.24 cluster and operator version is 1.3.6 NAME     
      ~~~                                  CPU(cores)   MEMORY(bytes)
      aide-worker-fileintegrity-266wv            300m         504Mi
      aide-worker-fileintegrity-hzz2d            300m         138Mi
      aide-worker-fileintegrity-vlpfc            244m         477Mi
      file-integrity-operator-79c7f495f9-6wltf   2m           23Mi
      ~~~
      
      I edit the resource limit, but still memory and cpu consumption continue to rise 
      ~~~
      $ oc get daemonsets.apps aide-worker-fileintegrity -o yaml 
       resources:
                limits:
                  cpu: 500m
                  memory: 800Mi
                requests:
                  cpu: 40m
                  memory: 40Mi
      NAME                                       CPU(cores)   MEMORY(bytes)
      aide-worker-fileintegrity-4nscl            249m         63Mi
      aide-worker-fileintegrity-rt9g9            413m         676Mi
      aide-worker-fileintegrity-w8kfm            447m         140Mi
      file-integrity-operator-77989587b6-zj57t   2m           27Mi
      
      NAME                                       CPU(cores)   MEMORY(bytes)
      aide-worker-fileintegrity-4nscl            142m         661Mi
      aide-worker-fileintegrity-rt9g9            390m         706Mi
      aide-worker-fileintegrity-w8kfm            333m         167Mi
      file-integrity-operator-77989587b6-zj57t   2m           27Mi
      
      

      Additional info:

      Memory and CPU consumption frequently rises close to the DaemonSet limit, It seems like regression of CMP-3662 bug.

              wenshen@redhat.com Vincent Shen
              rhn-support-aanasali Anas Ali
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: