Problem description

The compliance operator is currently not able to remediate fully some issues.

One example is the chrony server list as described in the bug below.

We should come up with a strategy to deal with remediations that only get fixed partially.

This could either get fixed by actually enabling a way for us to fix these kind of issues or by enabling a way to mark partial fixes so they don't confuse users.

Acceptance Criteria

• We have a repeatable and documented strategy dealing with "partial" fixes

Bug description

Copied originally from https://bugzilla.redhat.com/show_bug.cgi?id=1953277

Description of problem:
For some rules with label “compliance.openshift.io/automated-remediation=” cannot be auto remediated, while some other rules could be auto remediated without label“compliance.openshift.io/automated-remediation=”

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-24-175929

How reproducible:
Always

Steps to Reproduce:
1. Install compliance operator
2. Create scansettingbinding:
oc create -f - << EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
name: my-companys-compliance-requirements
profiles:

• name: ocp4-moderate
  apiGroup: compliance.openshift.io/v1alpha1
  kind: Profile
• name: rhcos4-moderate
  kind: Profile
  apiGroup: compliance.openshift.io/v1alpha1
  settingsRef:
  name: default-auto-apply
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1
  EOF
  3. Wait until cluster reboot is done.
  4. Check whether there are complianceremediations in “MissingDependencies” or outdated and trigger rescan

1. oc get complianceremediations | grep -i outdate
2. oc get complianceremediations | grep MissingDependencies
   rhcos4-moderate-master-configure-usbguard-auditbackend MissingDependencies
   rhcos4-moderate-master-service-usbguard-enabled MissingDependencies
   rhcos4-moderate-master-usbguard-allow-hid-and-hub MissingDependencies
   rhcos4-moderate-worker-configure-usbguard-auditbackend MissingDependencies
   rhcos4-moderate-worker-service-usbguard-enabled MissingDependencies
   rhcos4-moderate-worker-usbguard-allow-hid-and-hub MissingDependencies
3. oc annotate compliancescan ocp4-moderate rhcos4-moderate-master rhcos4-moderate-worker compliance.openshift.io/rescan=
   compliancescan.compliance.openshift.io/ocp4-moderate annotated
   compliancescan.compliance.openshift.io/rhcos4-moderate-master annotated
   compliancescan.compliance.openshift.io/rhcos4-moderate-worker annotated

5. Wait until cluster reboot is done.
6. Check whether there are checkresults in “MissingDependencies” or outdated and trigger rescan

Actual results:
1. The below rules with label “compliance.openshift.io/automated-remediation=” cannot actually be auto remediated.

1. cat rules.log
   rhcos4-moderate-master-chronyd-or-ntpd-set-maxpoll
   rhcos4-moderate-master-chronyd-or-ntpd-specify-multiple-servers
   rhcos4-moderate-master-chronyd-or-ntpd-specify-remote-server
   rhcos4-moderate-master-service-usbguard-enabled
   rhcos4-moderate-worker-chronyd-or-ntpd-set-maxpoll
   rhcos4-moderate-worker-chronyd-or-ntpd-specify-multiple-servers
   rhcos4-moderate-worker-chronyd-or-ntpd-specify-remote-server
   Rhcos4-moderate-worker-service-usbguard-enabled

1. for rules in `cat rules.log`; do oc get compliancecheckresults $rules --show-labels --no-headers; done
   rhcos4-moderate-master-chronyd-or-ntpd-set-maxpoll FAIL medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-master,compliance.openshift.io/suite=my-companys-compliance-requirements
   rhcos4-moderate-master-chronyd-or-ntpd-specify-multiple-servers FAIL medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-master,compliance.openshift.io/suite=my-companys-compliance-requirements
   rhcos4-moderate-master-chronyd-or-ntpd-specify-remote-server PASS medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=PASS,compliance.openshift.io/scan-name=rhcos4-moderate-master,compliance.openshift.io/suite=my-companys-compliance-requirements
   rhcos4-moderate-master-service-usbguard-enabled FAIL medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-master,compliance.openshift.io/suite=my-companys-compliance-requirements
   rhcos4-moderate-worker-chronyd-or-ntpd-set-maxpoll FAIL medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-worker,compliance.openshift.io/suite=my-companys-compliance-requirements
   rhcos4-moderate-worker-chronyd-or-ntpd-specify-multiple-servers FAIL medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-worker,compliance.openshift.io/suite=my-companys-compliance-requirements
   rhcos4-moderate-worker-chronyd-or-ntpd-specify-remote-server PASS medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=PASS,compliance.openshift.io/scan-name=rhcos4-moderate-worker,compliance.openshift.io/suite=my-companys-compliance-requirements
   rhcos4-moderate-worker-service-usbguard-enabled FAIL medium compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-worker,compliance.openshift.io/suite=my-companys-compliance-requirements

After remediation:
rhcos4-moderate-master-chronyd-or-ntpd-set-maxpoll FAIL medium
rhcos4-moderate-master-chronyd-or-ntpd-specify-multiple-servers FAIL medium
rhcos4-moderate-master-chronyd-or-ntpd-specify-remote-server FAIL medium
rhcos4-moderate-master-service-usbguard-enabled FAIL medium
rhcos4-moderate-worker-chronyd-or-ntpd-set-maxpoll FAIL medium
rhcos4-moderate-worker-chronyd-or-ntpd-specify-multiple-servers FAIL medium
rhcos4-moderate-worker-chronyd-or-ntpd-specify-remote-server FAIL medium
rhcos4-moderate-worker-service-usbguard-enabled FAIL medium

Take one check rhcos4-moderate-worker-chronyd-or-ntpd-set-maxpoll for example, there is remediation check available. However, it is not wroking as expected.

1. oc compliance fetch-fixes complianceremediations worker-scan-chronyd-or-ntpd-set-maxpoll
   Persisted compliance remediation fix to worker-scan-chronyd-or-ntpd-set-maxpoll.yaml
2. cat worker-scan-chronyd-or-ntpd-set-maxpoll.yaml
   —
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
   name: worker-scan-chronyd-or-ntpd-set-maxpoll
   spec:
   config:
   ignition:
   version: 3.1.0
   storage:
   files:

• contents:
  source: data:,%23%20Allow%20for%20extra%20configuration%20files.%20This%20is%20useful%0A%23%20for%20admins%20specifying%20their%20own%20NTP%20servers%0Ainclude%20/etc/chrony.d/%2A.conf%0A%0A%23%20Set%20chronyd%20as%20client-only.%0Aport%200%0A%0A%23%20Disable%20chronyc%20from%20the%20network%0Acmdport%200%0A%0A%23%20Record%20the%20rate%20at%20which%20the%20system%20clock%20gains/losses%20time.%0Adriftfile%20/var/lib/chrony/drift%0A%0A%23%20Allow%20the%20system%20clock%20to%20be%20stepped%20in%20the%20first%20three%20updates%0A%23%20if%20its%20offset%20is%20larger%20than%201%20second.%0Amakestep%201.0%203%0A%0A%23%20Enable%20kernel%20synchronization%20of%20the%20real-time%20clock%20%28RTC%29.%0Artcsync%0A%0A%23%20Enable%20hardware%20timestamping%20on%20all%20interfaces%20that%20support%20it.%0A%23hwtimestamp%20%2A%0A%0A%23%20Increase%20the%20minimum%20number%20of%20selectable%20sources%20required%20to%20adjust%0A%23%20the%20system%20clock.%0A%23minsources%202%0A%0A%23%20Allow%20NTP%20client%20access%20from%20local%20network.%0A%23allow%20192.168.0.0/16%0A%0A%23%20Serve%20time%20even%20if%20not%20synchronized%20to%20a%20time%20source.%0A%23local%20stratum%2010%0A%0A%23%20Require%20authentication%20%28nts%20or%20key%20option%29%20for%20all%20NTP%20sources.%0A%23authselectmode%20require%0A%0A%23%20Specify%20file%20containing%20keys%20for%20NTP%20authentication.%0Akeyfile%20/etc/chrony.keys%0A%0A%23%20Insert/delete%20leap%20seconds%20by%20slewing%20instead%20of%20stepping.%0A%23leapsecmode%20slew%0A%0A%23%20Get%20TAI-UTC%20offset%20and%20leap%20seconds%20from%20the%20system%20tz%20database.%0Aleapsectz%20right/UTC%0A%0A%23%20Specify%20directory%20for%20log%20files.%0Alogdir%20/var/log/chrony%0A%0A%23%20Select%20which%20information%20is%20logged.%0A%23log%20measurements%20statistics%20tracking
  mode: 420
  overwrite: true
  path: /etc/chrony.conf
• contents:
  source: data:,
  mode: 420
  overwrite: true
  path: /etc/chrony.d/.mco-keep

2. The below rules could be auto remediated but the label “compliance.openshift.io/automated-remediation=” is missing for these rules
Before remediation:
#oc get compliancecheckresults --show-labels | grep rule-order
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-open-by-handle-at-rule-order FAIL medium compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-master,compliance.openshift.io/suite=my-companys-compliance-requirements
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-open-rule-order FAIL medium compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-master,compliance.openshift.io/suite=my-companys-compliance-requirements
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-openat-rule-order FAIL medium compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-master,compliance.openshift.io/suite=my-companys-compliance-requirements
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-open-by-handle-at-rule-order FAIL medium compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-worker,compliance.openshift.io/suite=my-companys-compliance-requirements
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-open-rule-order FAIL medium compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-worker,compliance.openshift.io/suite=my-companys-compliance-requirements
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-openat-rule-order FAIL medium compliance.openshift.io/check-severity=medium,compliance.openshift.io/check-status=FAIL,compliance.openshift.io/scan-name=rhcos4-moderate-worker,compliance.openshift.io/suite=my-companys-compliance-requirements

After remediation:
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-open-by-handle-at-rule-order PASS medium
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-open-rule-order PASS medium
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-openat-rule-order PASS medium
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-open-by-handle-at-rule-order PASS medium
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-open-rule-order PASS medium
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-openat-rule-order PASS medium

Expected results:
1. The below rules with “compliance.openshift.io/automated-remediation=” label should be auto remediated as expected
rhcos4-moderate-master-chronyd-or-ntpd-set-maxpoll
rhcos4-moderate-master-chronyd-or-ntpd-specify-multiple-servers
rhcos4-moderate-master-chronyd-or-ntpd-specify-remote-server
rhcos4-moderate-master-service-usbguard-enabled
rhcos4-moderate-worker-chronyd-or-ntpd-set-maxpoll
rhcos4-moderate-worker-chronyd-or-ntpd-specify-multiple-servers
rhcos4-moderate-worker-chronyd-or-ntpd-specify-remote-server
Rhcos4-moderate-worker-service-usbguard-enabled
2. The below should have “compliance.openshift.io/automated-remediation=” label as could be auto remediated
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-open-by-handle-at-rule-order
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-open-rule-order
rhcos4-moderate-master-audit-rules-unsuccessful-file-modification-openat-rule-order
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-open-by-handle-at-rule-order
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-open-rule-order
rhcos4-moderate-worker-audit-rules-unsuccessful-file-modification-openat-rule-order