Description of problem:

This is related to the ODF BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2315619

A ConfigurationPolicy vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e from the namespace openshift-operators reported Compliance errors as the required configuration was not delivered for over ~18 days.

The errors initially (first few minutes) stem from the fact that the secret that was to be found in the openshift-dr-system namespace on the ManagedCluster was not delivered yet, and subsequently the target namespace that the secret needed to be "copied" into was not created.

Once the above were present, it still took over ~18 days for the policy to reconcile successfully.

Version-Release number of selected component (if applicable):

ACM - 2.12.0-79
OCP - 4.17.0-0.nightly-2024-09-12-145503

How reproducible:

Occurred once

Steps to Reproduce:

Ideally setting up initial DRPolicy for ODF disaster recovery setup should reproduce the issue, but that is a long pole. It maybe better to simulate it in case some other issues are suspected (see additional info)

Attached logs from (hopefully) relevant open-cluster-management pods.

Actual results:

ConfigurationPolicy was compliant only in ~18 days

Expected results:

Expected policy to be compliant once required resources were present on the ManagedCluster (in order of minutes)

Additional info:

Brief overview of the flow of ODF DR Policies in this regard:

• There is a first policy created, that is responsible to get the secret from the hub "openshift-operators" namespace into the ManagedCluster "openshift-dr-system" namespace
• There is a second policy that is responsible to create a secret in the "openshift-adp" namespace, based on the secret from the "openshift-dr-system" namespace

1. The second policy was created at the hub at "2024-09-16T07:44:21Z"
2. The secret was created in the openshift-dr-system namespace on the ManagedCluster at "2024-09-16T07:44:22Z"
3. The "openshift-adp" namespace was created on the ManagedCluster at "2024-09-16T07:48:50Z"
4. The secret was finally present in the "openshift-adp" namespace on the ManagedCluster at "2024-10-03T05:22:05Z"

Resource metadata:

Collected resources hub:

• (first policy) Secret propagation policy for a hub secret to the ManagedCluster (policy: d83ec4cb5be7c8c43f3cd41084f8d034b173c1e)
• Hub secret for the above (secret: d83ec4cb5be7c8c43f3cd41084f8d034b173c1e)
• (second policy) Hub policy for secret propogation of above secret from one namespace into another on the ManagedCluster (policy: vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e)
• Above policy on hub from the ManagedCluster namespace (policy: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e)

Collected resources ManagedCluster:

• Secret propagated as per first policy (secret: d83ec4cb5be7c8c43f3cd41084f8d034b173c1e)
• Policy propagated to the ManagedCluster cluster namespace "openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e"
• "openshift-adp" namespace (for creation time)
• Finale secret in "openshift-adp" namespace delivered as per policy (for creation timestamp)

On hub cluster
$ oc get policy -n openshift-operators d83ec4cb5be7c8c43f3cd41084f8d034b173c1e -o yaml
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  annotations:
    policy.open-cluster-management.io/trigger-update: "3194661"
  creationTimestamp: "2024-09-16T07:44:21Z"
  generation: 1
  name: d83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  namespace: openshift-operators
  resourceVersion: "3194753"
  uid: 2c86be94-b682-4f7f-b18d-d39faa1293a0
spec:
  disabled: false
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        creationTimestamp: null
        name: cfg-policy-d83ec4cb5be7c8c43f3cd41084f8d034b173c1e
      spec:
        evaluationInterval: {}
        namespaceSelector: {}
        object-templates:
        - complianceType: Musthave
          objectDefinition:
            apiVersion: v1
            data:
              AWS_ACCESS_KEY_ID: '{{hub fromSecret "openshift-operators" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e"
                "AWS_ACCESS_KEY_ID" hub}}'
              AWS_SECRET_ACCESS_KEY: '{{hub fromSecret "openshift-operators" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e"
                "AWS_SECRET_ACCESS_KEY" hub}}'
            kind: Secret
            metadata:
              creationTimestamp: null
              name: d83ec4cb5be7c8c43f3cd41084f8d034b173c1e
              namespace: openshift-dr-system
        remediationAction: Enforce
        severity: high
      status: {}
  remediationAction: Enforce
status:
  compliant: Compliant
  placement:
  - placementBinding: plbinding-d83ec4cb5be7c8c43f3cd41084f8d034b173c1e
    placementRule: plrule-d83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  status:
  - clustername: kmanohar-c1
    clusternamespace: kmanohar-c1
    compliant: Compliant
  - clustername: kmanohar-c2
    clusternamespace: kmanohar-c2
    compliant: Compliant

$ oc get secrets -n openshift-operators d83ec4cb5be7c8c43f3cd41084f8d034b173c1e -o yaml
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: YzAwa1puVmJnWnZkTnJubWVUb2c=
  AWS_SECRET_ACCESS_KEY: K0VYQ25HM3NqVzJRNUZFL1dXcjFXaldPem5hWTUvaWU3dDVmL2kvWg==
kind: Secret
metadata:
  creationTimestamp: "2024-09-16T07:43:38Z"
  finalizers:
  - drpolicies.ramendr.openshift.io/policy-protection
  - drpolicies.ramendr.openshift.io/policy-protection-velero
  labels:
    multicluster.odf.openshift.io/created-by: mirrorpeersecret
  name: d83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  namespace: openshift-operators
  resourceVersion: "3194661"
  uid: 16c0ba7a-44dc-43de-a2f3-ec53ae338ebe
type: Opaque

$ oc get policy -n openshift-operators vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e -o yaml
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  annotations:
    policy.open-cluster-management.io/trigger-update: "3194661"
  creationTimestamp: "2024-09-16T07:44:21Z"
  generation: 2
  name: vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  namespace: openshift-operators
  resourceVersion: "23388623"
  uid: 48910c01-0425-41a7-8e7a-6db6e5dfa0f6
spec:
  disabled: false
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        creationTimestamp: null
        name: cfg-policy-vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
      spec:
        evaluationInterval: {}
        namespaceSelector: {}
        object-templates:
        - complianceType: Musthave
          objectDefinition:
            apiVersion: v1
            data:
              ramengenerated: '{{ (printf "[default]\n  aws_access_key_id = %s\n  aws_secret_access_key
                = %s\n" ((lookup "v1" "Secret" "openshift-dr-system" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e").data.AWS_ACCESS_KEY_ID
                | base64dec) ((lookup "v1" "Secret" "openshift-dr-system" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e").data.AWS_SECRET_ACCESS_KEY
                | base64dec)) | base64enc }}'
            kind: Secret
            metadata:
              creationTimestamp: null
              name: vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
              namespace: openshift-adp
        remediationAction: Enforce
        severity: high
      status: {}
  remediationAction: enforce
status:
  compliant: Compliant
  placement:
  - placementBinding: plbinding-vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
    placementRule: plrule-vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  status:
  - clustername: kmanohar-c1
    clusternamespace: kmanohar-c1
    compliant: Compliant
  - clustername: kmanohar-c2
    clusternamespace: kmanohar-c2
    compliant: Compliant

$ oc get policy -n kmanohar-c1 openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e -o yaml
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
  creationTimestamp: "2024-09-16T07:44:22Z"
  generation: 2
  labels:
    policy.open-cluster-management.io/cluster-name: kmanohar-c1
    policy.open-cluster-management.io/cluster-namespace: kmanohar-c1
    policy.open-cluster-management.io/root-policy: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  name: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  namespace: kmanohar-c1
  resourceVersion: "23388765"
  uid: 689c8cf7-3dbb-4952-93a7-0bc4b15502d3
spec:
  disabled: false
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        creationTimestamp: null
        name: cfg-policy-vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
      spec:
        evaluationInterval: {}
        namespaceSelector: {}
        object-templates:
        - complianceType: Musthave
          objectDefinition:
            apiVersion: v1
            data:
              ramengenerated: '{{ (printf "[default]\n  aws_access_key_id = %s\n  aws_secret_access_key
                = %s\n" ((lookup "v1" "Secret" "openshift-dr-system" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e").data.AWS_ACCESS_KEY_ID
                | base64dec) ((lookup "v1" "Secret" "openshift-dr-system" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e").data.AWS_SECRET_ACCESS_KEY
                | base64dec)) | base64enc }}'
            kind: Secret
            metadata:
              creationTimestamp: null
              name: vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
              namespace: openshift-adp
        remediationAction: Enforce
        severity: high
      status: {}
  remediationAction: enforce
status:
  compliant: Compliant
  details:
  - compliant: Compliant
    history:
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17fad9d9730c4fe2
      lastTimestamp: "2024-10-03T05:22:15Z"
      message: Compliant; notification - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        found as specified in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17fad9d7208a655f
      lastTimestamp: "2024-10-03T05:22:05Z"
      message: Compliant; notification - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        was created successfully in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17fad9d71f91d0ba
      lastTimestamp: "2024-10-03T05:22:05Z"
      message: NonCompliant; violation - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        not found in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17f5a9bf7d5f0f50
      lastTimestamp: "2024-09-16T07:44:32Z"
      message: 'NonCompliant; violation - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        in namespace openshift-adp is missing, and cannot be created, reason: `namespaces
        "openshift-adp" not found`'
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17f5a9bf7cbb5642
      lastTimestamp: "2024-09-16T07:44:32Z"
      message: NonCompliant; violation - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        not found in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17f5a9bd42e538ae
      lastTimestamp: "2024-09-16T07:44:22Z"
      message: 'NonCompliant; violation - failed to resolve the template {"apiVersion":"v1","data":{"ramengenerated":"{{
        (printf \"[default]
n  aws_access_key_id = %s
n  aws_secret_access_key =
        %s\\n\" ((lookup \"v1\" \"Secret\" \"openshift-dr-system\" \"d83ec4cb5be7c8c43f3cd41084f8d034b173c1e\").data.AWS_ACCESS_KEY_ID
        | base64dec) ((lookup \"v1\" \"Secret\" \"openshift-dr-system\" \"d83ec4cb5be7c8c43f3cd41084f8d034b173c1e\").data.AWS_SECRET_ACCESS_KEY
        | base64dec)) | base64enc }}"},"kind":"Secret","metadata":{"creationTimestamp":null,"name":"vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e","namespace":"openshift-adp"}}:
        template: tmpl:3:213: executing "tmpl" at <base64dec>: invalid value; expected
        string'
    templateMeta:
      creationTimestamp: null
      name: cfg-policy-vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e

On ManagedCluster kmanohar-c1
$ oc get secrets -n openshift-dr-system d83ec4cb5be7c8c43f3cd41084f8d034b173c1e -o yaml
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: YzAwa1puVmJnWnZkTnJubWVUb2c=
  AWS_SECRET_ACCESS_KEY: K0VYQ25HM3NqVzJRNUZFL1dXcjFXaldPem5hWTUvaWU3dDVmL2kvWg==
kind: Secret
metadata:
  creationTimestamp: "2024-09-16T07:44:22Z"
  name: d83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  namespace: openshift-dr-system
  resourceVersion: "2888967"
  uid: 7ee858af-cfbb-43fc-9c58-267e4a25405d
type: Opaque

$ oc get policy -n kmanohar-c1 openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e -o yaml
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
  creationTimestamp: "2024-09-16T07:44:22Z"
  generation: 2
  labels:
    policy.open-cluster-management.io/cluster-name: kmanohar-c1
    policy.open-cluster-management.io/cluster-namespace: kmanohar-c1
    policy.open-cluster-management.io/root-policy: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  name: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  namespace: kmanohar-c1
  resourceVersion: "22826376"
  uid: 7b2592b6-44cc-49e3-9f51-c31ea87cbd55
spec:
  disabled: false
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        creationTimestamp: null
        name: cfg-policy-vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
      spec:
        evaluationInterval: {}
        namespaceSelector: {}
        object-templates:
        - complianceType: Musthave
          objectDefinition:
            apiVersion: v1
            data:
              ramengenerated: '{{ (printf "[default]\n  aws_access_key_id = %s\n  aws_secret_access_key
                = %s\n" ((lookup "v1" "Secret" "openshift-dr-system" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e").data.AWS_ACCESS_KEY_ID
                | base64dec) ((lookup "v1" "Secret" "openshift-dr-system" "d83ec4cb5be7c8c43f3cd41084f8d034b173c1e").data.AWS_SECRET_ACCESS_KEY
                | base64dec)) | base64enc }}'
            kind: Secret
            metadata:
              creationTimestamp: null
              name: vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
              namespace: openshift-adp
        remediationAction: Enforce
        severity: high
      status: {}
  remediationAction: enforce
status:
  compliant: Compliant
  details:
  - compliant: Compliant
    history:
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17fad9d9730c4fe2
      lastTimestamp: "2024-10-03T05:22:15Z"
      message: Compliant; notification - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        found as specified in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17fad9d7208a655f
      lastTimestamp: "2024-10-03T05:22:05Z"
      message: Compliant; notification - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        was created successfully in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17fad9d71f91d0ba
      lastTimestamp: "2024-10-03T05:22:05Z"
      message: NonCompliant; violation - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        not found in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17f5a9bf7d5f0f50
      lastTimestamp: "2024-09-16T07:44:32Z"
      message: 'NonCompliant; violation - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        in namespace openshift-adp is missing, and cannot be created, reason: `namespaces
        "openshift-adp" not found`'
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17f5a9bf7cbb5642
      lastTimestamp: "2024-09-16T07:44:32Z"
      message: NonCompliant; violation - secrets [vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e]
        not found in namespace openshift-adp
    - eventName: openshift-operators.vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e.17f5a9bd42e538ae
      lastTimestamp: "2024-09-16T07:44:22Z"
      message: 'NonCompliant; violation - failed to resolve the template {"apiVersion":"v1","data":{"ramengenerated":"{{
        (printf \"[default]
n  aws_access_key_id = %s
n  aws_secret_access_key =
        %s\\n\" ((lookup \"v1\" \"Secret\" \"openshift-dr-system\" \"d83ec4cb5be7c8c43f3cd41084f8d034b173c1e\").data.AWS_ACCESS_KEY_ID
        | base64dec) ((lookup \"v1\" \"Secret\" \"openshift-dr-system\" \"d83ec4cb5be7c8c43f3cd41084f8d034b173c1e\").data.AWS_SECRET_ACCESS_KEY
        | base64dec)) | base64enc }}"},"kind":"Secret","metadata":{"creationTimestamp":null,"name":"vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e","namespace":"openshift-adp"}}:
        template: tmpl:3:213: executing "tmpl" at <base64dec>: invalid value; expected
        string'
    templateMeta:
      creationTimestamp: null
      name: cfg-policy-vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e

$ oc get ns openshift-adp -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c28,c7
    openshift.io/sa.scc.supplemental-groups: 1000770000/10000
    openshift.io/sa.scc.uid-range: 1000770000/10000
  creationTimestamp: "2024-09-16T07:48:50Z"
  labels:
    kubernetes.io/metadata.name: openshift-adp
    olm.operatorgroup.uid/ff2a766a-5527-46f0-8e34-14f9d8c64c61: ""
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/audit-version: v1.24
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged
    pod-security.kubernetes.io/warn-version: v1.24
    security.openshift.io/scc.podSecurityLabelSync: "true"
  name: openshift-adp
  resourceVersion: "2893847"
  uid: bae03b81-0d78-4015-867b-53f5a9c52199
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

$ oc get secrets -n openshift-adp vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e -o yaml
apiVersion: v1
data:
  ramengenerated: W2RlZmF1bHRdCiAgYXdzX2FjY2Vzc19rZXlfaWQgPSBjMDBrWm5WYmdadmROcm5tZVRvZwogIGF3c19zZWNyZXRfYWNjZXNzX2tleSA9ICtFWENuRzNzalcyUTVGRS9XV3IxV2pXT3puYVk1L2llN3Q1Zi9pL1oK
kind: Secret
metadata:
  creationTimestamp: "2024-10-03T05:22:05Z"
  name: vd83ec4cb5be7c8c43f3cd41084f8d034b173c1e
  namespace: openshift-adp
  resourceVersion: "22826225"
  uid: 9c1be55c-55d2-498c-921e-b2f9c6eb0654
type: Opaque