-
Bug
-
Resolution: Done
-
Undefined
-
None
-
4.10
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Important
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem: Post compliance scan in Azure cluster, the MCP gets paused and does not un-pause to complete the operation. There appears to be a previous bug (https://bugzilla.redhat.com/show_bug.cgi?id=1975358) that resulted in (https://access.redhat.com/solutions/6136562) which eluded to upgrading the compliance operator, but the CU is using the v1.0.0 version, so it should work.
Version-Release number of selected component (if applicable): v1.0.0
How reproducible: Consistent
Steps to Reproduce:{code:none}
1. Change scan setting / scan setting binding
2.
3.
Actual results:
$ oc get compliancescan
NAME PHASE RESULT
ocp4-cis DONE NON-COMPLIANT
ocp4-cis-node-master DONE INCONSISTENT
ocp4-cis-node-worker AGGREGATING NOT-AVAILABLE
$
And we also see all the app pods are recycling.
Expected results: Change is applied with Phase at "Done" with a decision on RESULT field.
Additional info: SF Related case 03551835
inspects of the openshift-compliance namespace along with must-gathers of the environment can be found here.
In the last inspect, I can see the compliance operator "pausing" the pool, but I don't see the subsequent attempts to un-pause it;
$ omg logs -n openshift-compliance pod/compliance-operator-54cdfd9858-2wt2j | grep -i paus
2023-07-10T22:17:52.066596533Z
2023-07-11T01:00:09.005821947Z
{"level":"info","ts":"2023-07-11T01:00:09.005Z","logger":"suitectrl","msg":"Pausing pool","Request.Namespace":"openshift-compliance","Request.Name":"ocp4-cis-cibc","MachineConfigPool.Name":"master"}These logs can also be found in the case. Below is the related logs to the events above;
2023-07-10T22:17:46.300577073Z I0710 22:17:46.300510 1 request.go:690] Waited for 1.018769376s due to client-side throttling, not priority and fairness, request: GET:https://10.101.96.1:443/apis/dynatrace.com/v1beta1?timeout=32s
2023-07-11T01:00:58.715565548Z
2023-07-12T01:00:09.912034185Z
{"level":"error","ts":"2023-07-12T01:00:09.911Z","logger":"suitectrl","msg":"Could not update scan status","Request.Namespace":"openshift-compliance","Request.Name":"ocp4-cis-cibc","error":"Operation cannot be fulfilled on compliancesuites.compliance.openshift.io \"ocp4-cis-cibc\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).reconcileScanStatus\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:293\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).reconcileScans\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:270\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).Reconcile\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}2023-07-12T01:00:09.912034185Z
{"level":"error","ts":"2023-07-12T01:00:09.912Z","logger":"suitectrl","msg":"Retriable error","Request.Namespace":"openshift-compliance","Request.Name":"ocp4-cis-cibc","error":"Operation cannot be fulfilled on compliancesuites.compliance.openshift.io \"ocp4-cis-cibc\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ComplianceAsCode/compliance-operator/pkg/controller/common.ReturnWithRetriableError\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/common/errors.go:117\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).Reconcile\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:182\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}