Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-74931

Configmap openshift-controller-manager-config is getting updated constantly by HCP Control Plane Operator

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • In Progress
    • Bug Fix
    • Hide
      Cause: The Hosted Cluster Config Operator (HCCO) contained legacy logic that modified the openshift-controller-manager-config ConfigMap to disable the serviceaccount-pull-secrets controller when the image registry managementState was set to Removed. In OCP 4.20+, CPOv2 took over management of this ConfigMap, but HCCO continued modifying it on every reconciliation cycle.
      Consequence: The openshift-controller-manager-config ConfigMap was updated by HCCO every minute, which triggered the openshift-controller-manager file observer to detect changes and restart pods. This caused constant openshift-controller-manager pod restarts approximately once per minute.
      Fix: Removed the OCM config update logic from HCCO entirely, since CPOv2 manages the openshift-controller-manager-config ConfigMap in OCP 4.20+. HCCO no longer modifies this ConfigMap.
      Result: The openshift-controller-manager pods no longer experience unnecessary restarts caused by constant ConfigMap updates from HCCO.
      Show
      Cause: The Hosted Cluster Config Operator (HCCO) contained legacy logic that modified the openshift-controller-manager-config ConfigMap to disable the serviceaccount-pull-secrets controller when the image registry managementState was set to Removed. In OCP 4.20+, CPOv2 took over management of this ConfigMap, but HCCO continued modifying it on every reconciliation cycle. Consequence: The openshift-controller-manager-config ConfigMap was updated by HCCO every minute, which triggered the openshift-controller-manager file observer to detect changes and restart pods. This caused constant openshift-controller-manager pod restarts approximately once per minute. Fix: Removed the OCM config update logic from HCCO entirely, since CPOv2 manages the openshift-controller-manager-config ConfigMap in OCP 4.20+. HCCO no longer modifies this ConfigMap. Result: The openshift-controller-manager pods no longer experience unnecessary restarts caused by constant ConfigMap updates from HCCO.
    • None
    • None
    • None
    • None

      Description of problem:

      Since OCP 4.20, openshift-controller-manager pods are getting affected by constant restarts due to the following events (1 restart per minute eventually):

       $ oc logs -n <ns> openshift-controller-manager-xxx -p | tail -n 3
      2026-01-22T14:04:36.637044417Z I0122 14:04:36.637003       1 observer_polling.go:120] Observed file "/etc/kubernetes/config/config.yaml" has been modified (old="xxx16a", new="xxxe615")
      2026-01-22T14:04:36.637044417Z W0122 14:04:36.637038       1 builder.go:160] Restart triggered because of file /etc/kubernetes/config/config.yaml was modified
      2026-01-22T14:04:36.637072934Z W0122 14:04:36.637057       1 leaderelection.go:84] leader election lost
         

      This seems to happen because control plane operator is updating the openshift controller manager configmap every minute:

        - apiVersion: v1
          fieldsType: FieldsV1
          fieldsV1:
            f:data:
              .: {}
              f:config.yaml: {}
            f:metadata:
              f:ownerReferences:
                .: {}
                k:{"uid":"xxx30f"}: {}
          manager: hypershift-controlplane-manager
          operation: Update
          time: "2026-01-23T08:47:53Z" <---
      
        - apiVersion: v1
          fieldsType: FieldsV1
          fieldsV1:
            f:data:
              .: {}
              f:config.yaml: {}
            f:metadata:
              f:ownerReferences:
                .: {}
                k:{"uid":"xxx30f"}: {}
          manager: hypershift-controlplane-manager
          operation: Update
          time: "2026-01-23T09:22:52Z" <---
       

      No content from OCM configuration is modified, only the configmap is updated and resourceVersion is updated. 

      There are not clear logs in the control plane operator on why it is being created or reconciled. 

      The restart trigger does not appear to be the cause, because the library-go/observer_polling.go identifies any changes in the configmap due to a hash difference, it decides to restart the pods which it is an expected behavior.

       

      Version-Release number of selected component (if applicable):

      4.20

      How reproducible:

          

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

          

      Expected results:

          

      Additional info:

          

              rh-ee-brcox Bryan Cox
              rhn-support-bgomes Bruno Gomes
              None
              None
              Jim Ma Jim Ma
              None
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: