Description of problem:

Deployed but failed to enforce compliance operator template on local-cluster of IBM Z. Checked governance framework addon log in ns "open-cluster-management-agent-addon", and found controller stuck at "policy_status_sync"

2026-02-09T17:04:59.100Z info policy-template-sync templatesync/template_sync.go:595 Policy template created successfully {"Request.Namespace": "local-cluster", "Request.Name": "default.test-compliance-operator", "template": "comp-operator-ns"}

2026-02-09T17:04:59.109Z info policy-status-sync statussync/policy_status_sync.go:142 Recalculating details for policy templates {"controller": "policy-status-sync", "controllerGroup": "policy.open-cluster-management.io", "controllerKind": "Policy", "Policy": {"name":"default.test-compliance-operator","namespace":"local-cluster"}, "namespace": "local-cluster", "name": "default.test-compliance-operator", "reconcileID": "1828a0e0-7077-46f5-8e44-1149f8617519", "HubNamespace": "local-cluster"}

2026-02-09T17:04:59.199Z info policy-status-sync statussync/policy_status_sync.go:543 status mismatch on managed, update it {"controller": "policy-status-sync", "controllerGroup": "policy.open-cluster-management.io", "controllerKind": "Policy", "Policy": {"name":"default.test-compliance-operator","namespace":"local-cluster"}, "namespace": "local-cluster", "name": "default.test-compliance-operator", "reconcileID": "1828a0e0-7077-46f5-8e44-1149f8617519", "HubNamespace": "local-cluster"}

2026-02-09T17:04:59.230Z info policy-template-sync templatesync/template_sync.go:595 Policy template created successfully {"Request.Namespace": "local-cluster", "Request.Name": "default.test-compliance-operator", "template": "operatorpolicy-comp-operator"}

2026-02-09T17:04:59.231Z info policy-status-sync statussync/policy_status_sync.go:572 status not in sync, update the hub {"controller": "policy-status-sync", "controllerGroup": "policy.open-cluster-management.io", "controllerKind": "Policy", "Policy": {"name":"default.test-compliance-operator","namespace":"local-cluster"}, "namespace": "local-cluster", "name": "default.test-compliance-operator", "reconcileID": "1828a0e0-7077-46f5-8e44-1149f8617519", "HubNamespace": "local-cluster"}

2026-02-09T17:04:59.238Z error policy-status-sync statussync/policy_status_sync.go:578 Failed to update policy status on hub {"controller": "policy-status-sync", "controllerGroup": "policy.open-cluster-management.io", "controllerKind": "Policy", "Policy": {"name":"default.test-compliance-operator","namespace":"local-cluster"}, "namespace": "local-cluster", "name": "default.test-compliance-operator", "reconcileID": "1828a0e0-7077-46f5-8e44-1149f8617519", "HubNamespace": "local-cluster", "error": "Operation cannot be fulfilled on policies.policy.open-cluster-management.io \"default.test-compliance-operator\": the object has been modified; please apply your changes to the latest version and try again"}

open-cluster-management.io/governance-policy-framework-addon/controllers/statussync.(*PolicyReconciler).updateStatuses

/go/src/github.com/open-cluster-management-io/governance-policy-framework-addon/controllers/statussync/policy_status_sync.go:578

open-cluster-management.io/governance-policy-framework-addon/controllers/statussync.(*PolicyReconciler).Reconcile

/go/src/github.com/open-cluster-management-io/governance-policy-framework-addon/controllers/statussync/policy_status_sync.go:151

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile

/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler

/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem

/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1

/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202

2026-02-09T17:04:59.238Z error controller/controller.go:353 Reconciler error {"controller": "policy-status-sync", "controllerGroup": "policy.open-cluster-management.io", "controllerKind": "Policy", "Policy": {"name":"default.test-compliance-operator","namespace":"local-cluster"}, "namespace": "local-cluster", "name": "default.test-compliance-operator", "reconcileID": "1828a0e0-7077-46f5-8e44-1149f8617519", "error": "Operation cannot be fulfilled on policies.policy.open-cluster-management.io \"default.test-compliance-operator\": the object has been modified; please apply your changes to the latest version and try again"}

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler

/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem

/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1

/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202

Version-Release number of selected component (if applicable):

ACM 2.16

How reproducible:

Always

Steps to Reproduce:

1. Deploy compliance operator template to local-cluster
3. ...

Actual results:

The policy failed to be enforced and no subscription, installplan, and operator are created

Expected results:

Compliance operator installed successfully

Additional info:

The hub environment has been heavily loaded. Not sure if the env issue can cause the error above. However, I was able to manually install the compliance operator from the OCP catalog and everything was installed successfully.