-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
file-integrity-operator-1.3.6, file-integrity-operator-1.3.7
-
False
-
-
False
-
-
-
Description of problem:
OpenShift 4.18.z with File Integrity Operator 1.3.6, the aide-worker-fileintegrity DaemonSet pods show continuous CPU and memory consumption growth during hourly AIDE scan cycles. Even when resource limits are increased, memory and CPU utilization continue to rise until the new limits are almost reached. The issue was reproduced in a clean lab environment with the exact same OCP and operator versions, confirming it is not customer-specific, and not caused by configuration.
Version-Release number of selected component (if applicable):
I have reproduced the issue in the following environments: - OpenShift versions tested: - 4.18.22 - 4.18.24 - FIO client versions tested: - 1.3.6 - 1.3.7
How reproducible:
Always
Steps to Reproduce:
1. Deploy OCP 4.18.22
2. Install File Integrity Operator 1.3.6
3. Apply FileIntegrity object with default settings
4. Observe resource usage:
oc adm top pods -n openshift-file-integrity
Actual results:
cluster(4.18.22) and operator(1.3.6) version
~~~
NAME CPU(cores) MEMORY(bytes)
aide-worker-fileintegrity-46cnt 161m 569Mi
aide-worker-fileintegrity-svcmp 89m 355Mi
aide-worker-fileintegrity-wcpg7 300m 286Mi
file-integrity-operator-8479f84bd6-st4lm 2m 32Mi
NAME CPU(cores) MEMORY(bytes)
aide-worker-fileintegrity-46cnt 300m 544Mi
aide-worker-fileintegrity-svcmp 96m 580Mi
aide-worker-fileintegrity-wcpg7 301m 105Mi
file-integrity-operator-8479f84bd6-st4lm 1m 31Mi
~~~
I upgraded the File Integrity Operator to the latest version (1.3.7) on same openshift 4.18.22 but still facing same high consumption of cpu and memory.
~~~
NAME CPU(cores) MEMORY(bytes)
aide-worker-fileintegrity-742xh 299m 526Mi
aide-worker-fileintegrity-7sqcd 296m 546Mi
aide-worker-fileintegrity-b2bjf 239m 397Mi
file-integrity-operator-77989587b6-zj57t 2m 26Mi
~~~
I have reproduce the issue on 4.18.24 cluster and operator version is 1.3.6 NAME
~~~ CPU(cores) MEMORY(bytes)
aide-worker-fileintegrity-266wv 300m 504Mi
aide-worker-fileintegrity-hzz2d 300m 138Mi
aide-worker-fileintegrity-vlpfc 244m 477Mi
file-integrity-operator-79c7f495f9-6wltf 2m 23Mi
~~~
I edit the resource limit, but still memory and cpu consumption continue to rise
~~~
$ oc get daemonsets.apps aide-worker-fileintegrity -o yaml
resources:
limits:
cpu: 500m
memory: 800Mi
requests:
cpu: 40m
memory: 40Mi
NAME CPU(cores) MEMORY(bytes)
aide-worker-fileintegrity-4nscl 249m 63Mi
aide-worker-fileintegrity-rt9g9 413m 676Mi
aide-worker-fileintegrity-w8kfm 447m 140Mi
file-integrity-operator-77989587b6-zj57t 2m 27Mi
NAME CPU(cores) MEMORY(bytes)
aide-worker-fileintegrity-4nscl 142m 661Mi
aide-worker-fileintegrity-rt9g9 390m 706Mi
aide-worker-fileintegrity-w8kfm 333m 167Mi
file-integrity-operator-77989587b6-zj57t 2m 27Mi
Additional info:
Memory and CPU consumption frequently rises close to the DaemonSet limit, It seems like regression of CMP-3662 bug.