Description of problem:

Customer is reporting that for a single cluster, some policies are showing as "No status" in the RHACM UI (Screenshots available here: https://redhat-internal.slack.com/archives/CU4QXLPQB/p1710498127974269?thread_ts=1710498108.008349&cid=CU4QXLPQB).

The Policies are not showing any "COMPLIANCE STATUS" on the command line either:

oc get policies -n example-cluster-prod
NAME                                                  REMEDIATION ACTION   COMPLIANCE STATE   AGE
policies-cd.example-cluster-definition-namespace          enforce                                 2d22h
policies-cd.example-cluster-definition         enforce              Compliant          16h
policies-cm.cluster-admins-policy                     enforce                                 2d22h

When reviewing the object, the "compliant" field in the status is missing:

status:
  details:
  - templateMeta:
      creationTimestamp: null
      name: example-cluster-definition-namespace 

However ConfigurationPolicies all have a compliancy state. We reviewed the following logs and were unable to find any error related to this issue:

• The Policy on the hub, in the managed cluster namespace.
• The Policy on the managed cluster.
• The ConfigurationPolicy(s) on the managed cluster.

Workaround was to change the policies in some way to convince it to send a new event. For example, customer could change the severity field - once they did that, the Policy showed as "Compliant" as expected.

Version-Release number of selected component (if applicable):

RHACM 2.8.5 (advanced-cluster-management.v2.8.5)

OpenShift Container Platform 4.12.45

How reproducible:

Only on customer side

Steps to Reproduce:

1. Create policies for multiple clusters
2. Wait for some time

Actual results:

Some Policies are showing up as "No status".

Expected results:

All Policies report a status.

Additional info:

• The problem in this case was limited to a single cluster, which points to the issue being with a component on the cluster itself
• Slack discussion: https://redhat-internal.slack.com/archives/CU4QXLPQB/p1710498108008349
• must-gather for both managed cluster and also hub cluster available in attached Support Case