Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-74872

OCP console operator is stuck in a reconciliation loop

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.22.0
    • Management Console
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • x86_64
    • QA
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      I am observing a reconciliation loop in the OpenShift Console Operator involving OCP 4.22.0-ec.1 and ODF v4.21.0-84.stable. After enabling the odf-console plugin,the Console Operator is stuck in a loop "syncing console-config configmap", resulting in the console pods restarting repeatedly every few minutes.   

      Version-Release number of selected component (if applicable):

      OCP: 4.22.0-ec.1
      ODF: v4.21.0-84.stable 

      How reproducible:

      Reproducible each time    

      Steps to Reproduce:

          1. Deploy OCP and install the ODF operators
          2. Enable odf-console plugin
          
          

      Actual results:

      The console-operator is stuck in a reconciliation loop and restarts console pods every few minutes.
      
      # oc get pods -n openshift-console                                                                                                                                             NAME                         READY   STATUS        RESTARTS   AGE                                                                                                                                                                          console-5d947bcbdd-4gsm9     1/1     Terminating   0          3m33s                                                                                                                                                                        console-5d947bcbdd-zj248     1/1     Running       0          3m33s                                                                                                                                                                        console-7cfc946c9b-tp2sl     0/1     Running       0          5s                                                                                                                                                                           console-7cfc946c9b-xdtgw     0/1     Pending       0          5s                                                                                                                                                                           downloads-5d659859f5-k8757   1/1     Running       0          2d22h                                                                                                                                                                        downloads-5d659859f5-rvfzv   1/1     Running       0          2d22h
      
      
      

       

      Expected results:

      The console pods should run stably.    

      Additional info:

      Console operator log:
      
      I0130 15:32:06.413640       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
      "", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/console-config -n openshift-console:
      cause by changes in data.console-config.yaml
      I0130 15:32:06.420483       1 apps.go:155] Deployment "openshift-console/console" changes: {"metadata":{"annotations":{"console.openshift.io/console-config-version":"5317571","operator.openshift.io/spec-hash":"ab47360b77d51a243bc0ff894
      acfc63fd5eefd10899262f56db3270f8cb94745"}},"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"console.openshift.io/console-config-version":"5317571"}},"spec":{"containers":[{"comm
      and":["/opt/bridge/bin/bridge","--public-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVers
      ion":"v1","fieldPath":"metadata.name"}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:422e91c6e6f420673a7a7ed480f8e1123fabbf0f276c50d39e49de0f70dfe514","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":
      {"command":["sleep","25"]}}},"livenessProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"
      name":"https","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory"
      :"100Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSecon
      ds":10,"successThreshold":1,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name"
      :"console-oauth-config","readOnly":true},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","na
      me":"trusted-ca-bundle","readOnly":true},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"co
      nsole-serving-cert"}},{"name":"console-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{
      "items":[{"key":"ca-bundle.crt","path":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
      I0130 15:32:06.427216       1 warnings.go:110] "Warning: spec.template.spec.nodeSelector[node-role.kubernetes.io/master]: use \"node-role.kubernetes.io/control-plane\" instead"
      E0130 15:32:06.427365       1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.22.0-ec.1, 2 replicas available
      I0130 15:32:06.427635       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
      "", FieldPath:""}): type: 'Normal' reason: 'DeploymentUpdated' Updated Deployment.apps/console -n openshift-console because it changed
      I0130 15:32:06.450639       1 status_controller.go:230] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"
      },{"lastTransitionTime":"2026-01-30T15:32:06Z","message":"SyncLoopRefreshProgressing: working toward version 4.22.0-ec.1, 2 replicas available","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitio
      nTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2026-01-30T08:14:45Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradea
      ble"},{"lastTransitionTime":"2026-01-27T16:53:37Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
      I0130 15:32:06.454436       1 apps.go:155] Deployment "openshift-console/console" changes: {"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"template":{"spec":{"containers":[{"command":["/opt/bridge/bin/bridge","--pu
      blic-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name
      "}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:422e91c6e6f420673a7a7ed480f8e1123fabbf0f276c50d39e49de0f70dfe514","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sleep","25"]}}},"livene
      ssProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"name":"https","protocol":"TCP"}],"re
      adinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory":"100Mi"}},"securityContext":{"allow
      PrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeout
      Seconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name":"console-oauth-config","readOnly":t
      rue},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","name":"trusted-ca-bundle","readOnly":t
      rue},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"console-serving-cert"}},{"name":"conso
      le-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{"items":[{"key":"ca-bundle.crt","pat
      h":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
      I0130 15:32:06.459548       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
      "", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from False to True ("SyncLoopRefreshProgressing: working toward version 4.22.0-ec.1, 2 replicas availabl
      e")
      E0130 15:32:06.461719       1 status.go:130] DeploymentSyncDegraded FailedApply Operation cannot be fulfilled on deployments.apps "console": the object has been modified; please apply your changes to the latest version and try again
      I0130 15:32:06.461807       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
      "", FieldPath:""}): type: 'Warning' reason: 'DeploymentUpdateFailed' Failed to update Deployment.apps/console -n openshift-console: Operation cannot be fulfilled on deployments.apps "console": the object has been modified; please apply
       your changes to the latest version and try again
      E0130 15:32:06.477349       1 base_controller.go:279] "Unhandled Error" err="ConsoleOperator reconciliation failed: Operation cannot be fulfilled on deployments.apps \"console\": the object has been modified; please apply your changes 
      to the latest version and try again"
      E0130 15:32:06.488808       1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.22.0-ec.1, 1 replicas available
      I0130 15:32:06.553503       1 status_controller.go:230] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"DeploymentSyncDegraded: Operation cannot be fulfilled on deployments.
      apps \"console\": the object has been modified; please apply your changes to the latest version and try again","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2026-01-30T15:32:06Z","message":"All is wel
      l","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2026-01-30T08:14:45Z"
      ,"message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2026-01-27T16:53:37Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
      I0130 15:32:06.822109       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
      "", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Degraded message changed from "All is well" to "DeploymentSyncDegraded: Operation cannot be fulfilled on deployments.apps \"
      console\": the object has been modified; please apply your changes to the latest version and try again",Progressing changed from True to False ("All is well")
      
      # oc get cm console-config -n openshift-console -o yaml
      apiVersion: v1
      data:
        console-config.yaml: |
          apiVersion: console.openshift.io/v1
          auth:
            authType: openshift
            clientID: console
            clientSecretFile: /var/oauth-config/clientSecret
            inactivityTimeoutSeconds: 3600
            oauthEndpointCAFile: /var/oauth-serving-cert/ca-bundle.crt
          clusterInfo:
            consoleBaseAddress: https://console-openshift-console.apps.hubcluster-hp.nokia-stamp705.bos2.lab
            controlPlaneTopology: HighlyAvailable
            masterPublicURL: https://api.hubcluster-hp.nokia-stamp705.bos2.lab:6443
            nodeArchitectures:
            - amd64
            nodeOperatingSystems:
            - linux
            releaseVersion: 4.22.0-ec.1
          contentSecurityPolicyEnabled: true
          customization:
            branding: ocp
            capabilities:
            - name: LightspeedButton
              visibility:
                state: Enabled
            - name: GettingStartedBanner
              visibility:
                state: Enabled
            developerCatalog:
              categories: null
              types:
                state: Enabled
            documentationBaseURL: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.21/
            perspectives:
            - id: dev
              visibility:
                state: Disabled
          i18nNamespaces:
          - plugin__networking-console-plugin
          - plugin__monitoring-plugin
          - plugin__mce
          - plugin__acm
          - plugin__logging-view-plugin
          kind: ConsoleConfig
          monitoringInfo:
            alertmanagerTenancyHost: alertmanager-main.openshift-monitoring.svc:9092
            alertmanagerUserWorkloadHost: alertmanager-main.openshift-monitoring.svc:9094
          plugins:
            acm: https://console-chart-console-v2.open-cluster-management.svc.cluster.local:3000/plugin/
            logging-view-plugin: https://logging.openshift-operators.svc.cluster.local:9443/
            mce: https://console-mce-console.multicluster-engine.svc.cluster.local:3000/plugin/
            monitoring-plugin: https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/
            networking-console-plugin: https://networking-console-plugin.openshift-network-console.svc.cluster.local:9443/
            odf-console: https://odf-console-service.openshift-storage.svc.cluster.local:9001/compatibility/
          pluginsOrder:
          - networking-console-plugin
          - monitoring-plugin
          - mce
          - acm
          - logging-view-plugin
          - odf-console
          providers: {}
          proxy:
            services:
            - authorize: true
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/mce/console/
              endpoint: https://console-mce-console.multicluster-engine.svc.cluster.local:3000
            - authorize: true
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/acm/console/
              endpoint: https://console-chart-console-v2.open-cluster-management.svc.cluster.local:3000
            - authorize: true
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/logging-view-plugin/backend/
              endpoint: https://logging-loki-gateway-http.openshift-logging.svc.cluster.local:8080
            - authorize: true
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/logging-view-plugin/korrel8r/
              endpoint: https://korrel8r.openshift-operators.svc.cluster.local:9443
            - authorize: true
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/odf-console/provider-proxy/
              endpoint: https://ux-backend-proxy.openshift-storage.svc.cluster.local:8888
            - authorize: true
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/odf-console/rosa-prometheus/
              endpoint: https://prometheus.openshift-storage.svc.cluster.local:9339
            - authorize: false
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/odf-console/s3/
              endpoint: https://s3.openshift-storage.svc.cluster.local:443
            - authorize: false
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/odf-console/iam/
              endpoint: https://iam.openshift-storage.svc.cluster.local:443
            - authorize: false
              caCertificate: ""
              consoleAPIPath: /api/proxy/plugin/odf-console/internalRgwS3/
              endpoint: https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:443
          servingInfo:
            bindAddress: https://[::]:8443
            certFile: /var/serving-cert/tls.crt
            keyFile: /var/serving-cert/tls.key
          session: {}
          telemetry:
            CLUSTER_ID: 0f3a49c2-6538-4b02-bc6d-f1719e4394af
            SEGMENT_API_HOST: console.redhat.com/connections/api/v1
            SEGMENT_JS_HOST: console.redhat.com/connections/cdn
            SEGMENT_PUBLIC_API_KEY: BnuS1RP39EmLQjP21ko67oDjhbl9zpNU
            TELEMETER_CLIENT_DISABLED: "true"
      kind: ConfigMap
      metadata:
        creationTimestamp: "2026-01-27T17:06:25Z"
        labels:
          app: console
        name: console-config
        namespace: openshift-console
        ownerReferences:
        - apiVersion: operator.openshift.io/v1
          controller: true
          kind: Console
          name: cluster
          uid: 955b03dc-d9f2-4dd2-97bb-2d6dcbc6d70f
        resourceVersion: "5353479"
        uid: fc9cecbb-65b9-4e94-8651-6c3ef7e0df91

              jhadvig@redhat.com Jakub Hadvig
              rh-ee-junyang Laura Yang
              None
              None
              YaDan Pei YaDan Pei
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: