Description:

File integrity not working correctly while removing and adding the directory in the node.

Reproduction steps:

• Environment
  ```
  file-integrity-operator.v1.3.3
  OC version: 4.15.0
  ```
  Testing Steps:

```
$ oc project
Using project "openshift-file-integrity" on server "https://api.sak415.lab.upshift.rdu2.redhat.com:6443".
```

• Verify that the File Integrity Operator is up and running:
  ```
  $ oc get deploy -n openshift-file-integrity
  NAME READY UP-TO-DATE AVAILABLE AGE
  file-integrity-operator 1/1 1 1 25h
  ```
• Check the file integrity instance created
  ````
  $ oc get fileintegrities
  NAME AGE
  worker-fileintegrity 25h
  ```
• Check all pods in OpenShift-file-integrity are in Running state
  ```
  $ oc get pods
  NAME READY STATUS RESTARTS AGE
  aide-worker-fileintegrity-4bg4k 1/1 Running 0 25h
  aide-worker-fileintegrity-tshbx 1/1 Running 0 25h
  aide-worker-fileintegrity-z7z4l 1/1 Running 0 25h
  file-integrity-operator-6f76cf869c-kv57c 0/1 Error 1 (25h ago) 25h
  file-integrity-operator-6f76cf869c-tnqz2 1/1 Running 0 21h
  ```
• Check the fileintegritynodestatus
  ```
  $ oc get fileintegritynodestatuses
  NAME NODE STATUS
  worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com worker-0.sak415.lab.upshift.rdu2.redhat.com Succeeded
  worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com worker-1.sak415.lab.upshift.rdu2.redhat.com Succeeded
  worker-fileintegrity-worker-2.sak415.lab.upshift.rdu2.redhat.com worker-2.sak415.lab.upshift.rdu2.redhat.com Succeeded
  ```
  All nodes have Succeeded status

Scenario I : Create a directory in the node `Worker1`

• Add a NEW directory sak in worker1 and check how FIO behaves

```
$ oc get fileintegritynodestatuses
NAME NODE STATUS
worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com worker-0.sak415.lab.upshift.rdu2.redhat.com Failed
worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com worker-1.sak415.lab.upshift.rdu2.redhat.com Succeeded
worker-fileintegrity-worker-2.sak415.lab.upshift.rdu2.redhat.com worker-2.sak415.lab.upshift.rdu2.redhat.com Succeeded

$ oc debug node/worker-1.sak415.lab.upshift.rdu2.redhat.com
Temporary namespace openshift-debug-fbzgw is created for debugging node...
Starting pod/worker-1sak415labupshiftrdu2redhatcom-debug-q6vcn ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.89.189
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-5.1# mkdir sak
mkdir: cannot create directory 'sak': Operation not permitted
sh-5.1# mkdir /etc/sak
sh-5.1# echo "hello" > /etc/sak/test1
sh-5.1# cat /etc/sak/test1
hello
sh-4.4# exit
exit

Removing debug pod .
```

• Checking the nodeStatus
  ```
  $ oc get fileintegritynodestatuses
  NAME NODE STATUS
  worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com worker-0.sak415.lab.upshift.rdu2.redhat.com Failed
  worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com worker-1.sak415.lab.upshift.rdu2.redhat.com Failed
  worker-fileintegrity-worker-2.sak415.lab.upshift.rdu2.redhat.com worker-2.sak415.lab.upshift.rdu2.redhat.com Succeeded

$ oc get events --field-selector reason=NodeIntegrityStatus
LAST SEEN TYPE REASON OBJECT MESSAGE
62m Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-0.sak415.lab.upshift.rdu2.redhat.com has changed! a:0,c:1,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com-failed
14m Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-0.sak415.lab.upshift.rdu2.redhat.com has changed! a:1,c:1,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com-failed
2m5s Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-1.sak415.lab.upshift.rdu2.redhat.com has changed! a:1,c:0,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
72s Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-1.sak415.lab.upshift.rdu2.redhat.com has changed! a:3,c:0,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
worker-1.sak415.lab.upshift.rdu2.redhat.com has Failed status
```

• Checking configmap of the failed node worker-1.sak415.lab.upshift.rdu2.redhat.com

````
$ oc describe fileintegritynodestatuses/worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com | grep -i "config" | tail -2
Result Config Map Name: aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
Result Config Map Namespace: openshift-file-integrity

$ oc describe cm aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
Name: aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
Namespace: openshift-file-integrity
Labels: file-integrity.openshift.io/node=worker-1.sak415.lab.upshift.rdu2.redhat.com
file-integrity.openshift.io/owner=worker-fileintegrity
file-integrity.openshift.io/result-log=
Annotations: file-integrity.openshift.io/files-added: 3
file-integrity.openshift.io/files-changed: 0
file-integrity.openshift.io/files-removed: 0

Data
====
integritylog:

Start timestamp: 2024-04-05 03:08:16 +0000 (AIDE 0.16)
AIDE found differences between database and filesystem!!

Summary:
Total number of entries: 32388
Added entries: 3
Removed entries: 0
Changed entries: 0

---------------------------------------------------
Added entries:
---------------------------------------------------

d++++++++++++++++: /hostroot/etc/sak
f++++++++++++++++: /hostroot/etc/sak/test1
f++++++++++++++++: /hostroot/root/.bash_history

---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------

/hostroot/etc/kubernetes/aide.db.gz
MD5 : FvOy0sDgSaC7Qsuy4LGZhw==
SHA1 : aZmihLGrIaMXPhzveXzaDut8C+8=
RMD160 : P9VYJBnOfOM7aM60RBHwsf1enX4=
TIGER : m9SwJkaVtrwfMagi9rNdd1vzaFwFGDEI
SHA256 : t59RnF8J6sJ+9yaTnePUYBNPiZwcR1p4
bQgxpDRm20I=
SHA512 : nnpdDbSO7EUGM6Mfu1tHMHu0dBXQXFnB
n4EXG3nkgc58M+trT+7I5uQUB2/tALCu
0Ip6r4BpF8t6ZAMIrRUdrQ==

End timestamp: 2024-04-05 03:08:47 +0000 (run time: 0m 31s)

BinaryData
====

Events: <none>

SCENARIO 2: Deleting director sak manually which was created in step2 in worker-1.sak415.lab.upshift.rdu2.redhat.com

````
$ oc get fileintegritynodestatuses
NAME NODE STATUS
worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com worker-0.sak415.lab.upshift.rdu2.redhat.com Failed
worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com worker-1.sak415.lab.upshift.rdu2.redhat.com Failed
worker-fileintegrity-worker-2.sak415.lab.upshift.rdu2.redhat.com worker-2.sak415.lab.upshift.rdu2.redhat.com Succeeded

$ oc debug node/worker-1.sak415.lab.upshift.rdu2.redhat.com
Temporary namespace openshift-debug-cf2mf is created for debugging node...
Starting pod/worker-1sak415labupshiftrdu2redhatcom-debug-zzj8g ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.89.189
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-5.1# ls /etc/sak
test1
sh-5.1# rm -rf /etc/sak
sh-5.1# ls /etc/sak
ls: cannot access '/etc/sak': No such file or directory
h-5.1# exit
exit
sh-4.4# exit
exit
```

• Checking Nodestatus again
  [sasakshi@sasakshi ~]$ oc get fileintegritynodestatuses
  NAME NODE STATUS
  worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com worker-0.sak415.lab.upshift.rdu2.redhat.com Failed
  worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com worker-1.sak415.lab.upshift.rdu2.redhat.com Failed
  worker-fileintegrity-worker-2.sak415.lab.upshift.rdu2.redhat.com worker-2.sak415.lab.upshift.rdu2.redhat.com Succeeded

• Checking config map of the failed Node
  ```
  [sasakshi@sasakshi ~]$ oc describe cm aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
  Name: aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
  Namespace: openshift-file-integrity
  Labels: file-integrity.openshift.io/node=worker-1.sak415.lab.upshift.rdu2.redhat.com
  file-integrity.openshift.io/owner=worker-fileintegrity
  file-integrity.openshift.io/result-log=
  Annotations: file-integrity.openshift.io/files-added: 1
  file-integrity.openshift.io/files-changed: 0
  file-integrity.openshift.io/files-removed: 0

Data
====
integritylog:

Start timestamp: 2024-04-05 03:32:25 +0000 (AIDE 0.16)
AIDE found differences between database and filesystem!!

Summary:
Total number of entries: 32386
Added entries: 1
Removed entries: 0
Changed entries: 0

---------------------------------------------------
Added entries:
---------------------------------------------------

f++++++++++++++++: /hostroot/root/.bash_history

---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------

/hostroot/etc/kubernetes/aide.db.gz
MD5 : FvOy0sDgSaC7Qsuy4LGZhw==
SHA1 : aZmihLGrIaMXPhzveXzaDut8C+8=
RMD160 : P9VYJBnOfOM7aM60RBHwsf1enX4=
TIGER : m9SwJkaVtrwfMagi9rNdd1vzaFwFGDEI
SHA256 : t59RnF8J6sJ+9yaTnePUYBNPiZwcR1p4
bQgxpDRm20I=
SHA512 : nnpdDbSO7EUGM6Mfu1tHMHu0dBXQXFnB
n4EXG3nkgc58M+trT+7I5uQUB2/tALCu
0Ip6r4BpF8t6ZAMIrRUdrQ==

End timestamp: 2024-04-05 03:32:58 +0000 (run time: 0m 33s)

BinaryData
====

Events: <none>
```

• Checking the events
  ````

95m Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-0.sak415.lab.upshift.rdu2.redhat.com has changed! a:0,c:1,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com-failed
47m Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-0.sak415.lab.upshift.rdu2.redhat.com has changed! a:1,c:1,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-0.sak415.lab.upshift.rdu2.redhat.com-failed
8m20s Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-1.sak415.lab.upshift.rdu2.redhat.com has changed! a:1,c:0,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
34m Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-1.sak415.lab.upshift.rdu2.redhat.com has changed! a:3,c:0,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed

Observations From SCENARIO2:

The events and failed config map report incorrect entries . No entries for removal are reported.
```
8m20s Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-1.sak415.lab.upshift.rdu2.redhat.com has changed! a:1,c:0,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
34m Warning NodeIntegrityStatus fileintegrity/worker-fileintegrity node worker-1.sak415.lab.upshift.rdu2.redhat.com has changed! a:3,c:0,r:0 log:openshift-file-integrity/aide-worker-fileintegrity-worker-1.sak415.lab.upshift.rdu2.redhat.com-failed
```

Actions Required:

• To check why entries are not reported correctly in the events and configmap in the SCENARIO 2 and is not Changing and keeping old value