Description of problem:
During ZTP, we reply TALO to create enforce policies automatically. Occasionally, TALO will think the common config-policy was compliant before starting, thus no enforce policy was created, causing ZTP to fail.

Version-Release number of selected component (if applicable):
4.10 TALO was used in this test

How reproducible:
Rare

Steps to Reproduce:
1. Trigger ZTP of a SNO spoke cluster
2. Check enforce policies created by CGU
3.

Actual results:

• no common config-policy enforce policy was created before TALO thinks it's already compliant.

Expected results:

• common config-policy is enforced via TALO

Additional info:

TALO status:

copiedPolicies:

• cnfde4-cnfde4-config-policy
• cnfde4-cnfde4-perf-policy
• cnfde4-common-subscriptions-policy
• cnfde4-group-du-sno-config-policy
• cnfde4-group-du-sno-validator-du-policy
  managedPoliciesCompliantBeforeUpgrade:
• common-config-policy
  managedPoliciesContent:
  cnfde4-config-policy: "null"
  cnfde4-perf-policy: "null"
  common-subscriptions-policy: '[ {"kind":"Subscription","name":"sriov-network-operator-subscription","namespace":"openshift-sriov-network-operator"}
  , {"kind":"Subscription","name":"ptp-operator-subscription","namespace":"openshift-ptp"}

  ,

  {"kind":"Subscription","name":"cluster-logging","namespace":"openshift-logging"}

  ,

  {"kind":"Subscription","name":"local-storage-operator","namespace":"openshift-local-storage"}

  ,

  {"kind":"Subscription","name":"sriov-fec-subscription","namespace":"vran-acceleration-operators"}

  ,

  {"kind":"Subscription","name":"amq7-interconnect-subscription","namespace":"amq-router"}

  ]'
  group-du-sno-config-policy: "null"
  group-du-sno-validator-du-policy: "null"

Policy history/status:

Parent policy under ztp-common namespace:
status:
compliant: NonCompliant
placement:

• placementBinding: common-placementbinding
  placementRule: common-placementrules
  status:
• clustername: cnfde4
  clusternamespace: cnfde4
  compliant: NonCompliant

Child policy under cnfde4 namespace:
status:
compliant: NonCompliant
details:

• compliant: NonCompliant
  history:
• eventName: ztp-common.common-config-policy.16f2b85ab86cbf82
  lastTimestamp: "2022-05-26T17:38:32Z"
  message: 'NonCompliant; violation - catalogsources not found: [redhat-operators]
  in namespace openshift-marketplace found but not as specified; violation -
  operatorhubs not found: [cluster] found but not as specified; violation -
  configmaps not found: [cluster-monitoring-config] in namespace openshift-monitoring
  missing; violation - catalogsources not found: [certified-operators] in namespace
  openshift-marketplace found but not as specified; violation - catalogsources
  not found: [amq-operators] in namespace openshift-marketplace missing'
  templateMeta:
  creationTimestamp: null
  name: common-config-policy-config

Part of TALO logs, not sure if the ERROR has anything to do with it:
2022-05-26T13:30:19.668Z ERROR controller-runtime.manager.controller.clustergroupupgrade Reconciler error

{"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "cnfde4", "namespace": "ztp-install", "error": "Operation cannot be fulfilled on clustergroupupgrades.ran.openshift.io \"cnfde4\": the object has been modified; please apply your changes to the latest ve rsion and try again"}

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214
2022-05-26T13:30:19.668Z INFO controllers.ClusterGroupUpgrade Start reconciling CGU

{"name": "cnfde4", "version": "19519692"}

2022-05-26T13:30:19.668Z INFO controllers.ClusterGroupUpgrade [getClusterBySelectors]

{"clustersBySelector": []}

2022-05-26T13:30:19.668Z INFO controllers.ClusterGroupUpgrade [getClustersBySelectors]

{"clusterNames": ["cnfde4"]}

2022-05-26T13:30:19.668Z INFO controllers.ClusterGroupUpgrade [getClusterBySelectors]

{"clustersBySelector": []}

2022-05-26T13:30:19.668Z INFO controllers.ClusterGroupUpgrade [getClustersBySelectors]

{"clusterNames": ["cnfde4"]}

2022-05-26T13:30:19.668Z INFO controllers.ClusterGroupUpgrade [doManagedPoliciesExist] {"policyMap": {"cnfde4-config-policy":"ztp-site","cnfde4-perf-policy":"ztp-site","common-config-policy":"ztp-common","common-subscriptions-policy":"ztp-common","group-du-sno-config-policy":"ztp-group","group-du-sno-validator-du-policy":"ztp-group"}}
2022-05-26T13:30:19.672Z INFO controllers.ClusterGroupUpgrade [getClusterBySelectors]

{"clustersBySelector": []}

2022-05-26T13:30:19.672Z INFO controllers.ClusterGroupUpgrade [getClustersBySelectors]

{"clusterNames": ["cnfde4"]}

2022-05-26T13:30:19.672Z INFO controllers.ClusterGroupUpgrade [getClustersNonCompliantWithPolicy]

{"policy: ": "common-config-policy", "clusters: ": []}