-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.22.0
-
None
Description of problem:
I am observing a reconciliation loop in the OpenShift Console Operator involving OCP 4.22.0-ec.1 and ODF v4.21.0-84.stable. After enabling the odf-console plugin,the Console Operator is stuck in a loop "syncing console-config configmap", resulting in the console pods restarting repeatedly every few minutes.
Version-Release number of selected component (if applicable):
OCP: 4.22.0-ec.1 ODF: v4.21.0-84.stable
How reproducible:
Reproducible each time
Steps to Reproduce:
1. Deploy OCP and install the ODF operators
2. Enable odf-console plugin
Actual results:
The console-operator is stuck in a reconciliation loop and restarts console pods every few minutes. # oc get pods -n openshift-console NAME READY STATUS RESTARTS AGE console-5d947bcbdd-4gsm9 1/1 Terminating 0 3m33s console-5d947bcbdd-zj248 1/1 Running 0 3m33s console-7cfc946c9b-tp2sl 0/1 Running 0 5s console-7cfc946c9b-xdtgw 0/1 Pending 0 5s downloads-5d659859f5-k8757 1/1 Running 0 2d22h downloads-5d659859f5-rvfzv 1/1 Running 0 2d22h
Expected results:
The console pods should run stably.
Additional info:
Console operator log:
I0130 15:32:06.413640 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/console-config -n openshift-console:
cause by changes in data.console-config.yaml
I0130 15:32:06.420483 1 apps.go:155] Deployment "openshift-console/console" changes: {"metadata":{"annotations":{"console.openshift.io/console-config-version":"5317571","operator.openshift.io/spec-hash":"ab47360b77d51a243bc0ff894
acfc63fd5eefd10899262f56db3270f8cb94745"}},"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"console.openshift.io/console-config-version":"5317571"}},"spec":{"containers":[{"comm
and":["/opt/bridge/bin/bridge","--public-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVers
ion":"v1","fieldPath":"metadata.name"}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:422e91c6e6f420673a7a7ed480f8e1123fabbf0f276c50d39e49de0f70dfe514","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":
{"command":["sleep","25"]}}},"livenessProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"
name":"https","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory"
:"100Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSecon
ds":10,"successThreshold":1,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name"
:"console-oauth-config","readOnly":true},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","na
me":"trusted-ca-bundle","readOnly":true},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"co
nsole-serving-cert"}},{"name":"console-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{
"items":[{"key":"ca-bundle.crt","path":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
I0130 15:32:06.427216 1 warnings.go:110] "Warning: spec.template.spec.nodeSelector[node-role.kubernetes.io/master]: use \"node-role.kubernetes.io/control-plane\" instead"
E0130 15:32:06.427365 1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.22.0-ec.1, 2 replicas available
I0130 15:32:06.427635 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'DeploymentUpdated' Updated Deployment.apps/console -n openshift-console because it changed
I0130 15:32:06.450639 1 status_controller.go:230] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"
},{"lastTransitionTime":"2026-01-30T15:32:06Z","message":"SyncLoopRefreshProgressing: working toward version 4.22.0-ec.1, 2 replicas available","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitio
nTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2026-01-30T08:14:45Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradea
ble"},{"lastTransitionTime":"2026-01-27T16:53:37Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0130 15:32:06.454436 1 apps.go:155] Deployment "openshift-console/console" changes: {"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"template":{"spec":{"containers":[{"command":["/opt/bridge/bin/bridge","--pu
blic-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name
"}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:422e91c6e6f420673a7a7ed480f8e1123fabbf0f276c50d39e49de0f70dfe514","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sleep","25"]}}},"livene
ssProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"name":"https","protocol":"TCP"}],"re
adinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory":"100Mi"}},"securityContext":{"allow
PrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeout
Seconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name":"console-oauth-config","readOnly":t
rue},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","name":"trusted-ca-bundle","readOnly":t
rue},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"console-serving-cert"}},{"name":"conso
le-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{"items":[{"key":"ca-bundle.crt","pat
h":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
I0130 15:32:06.459548 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from False to True ("SyncLoopRefreshProgressing: working toward version 4.22.0-ec.1, 2 replicas availabl
e")
E0130 15:32:06.461719 1 status.go:130] DeploymentSyncDegraded FailedApply Operation cannot be fulfilled on deployments.apps "console": the object has been modified; please apply your changes to the latest version and try again
I0130 15:32:06.461807 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Warning' reason: 'DeploymentUpdateFailed' Failed to update Deployment.apps/console -n openshift-console: Operation cannot be fulfilled on deployments.apps "console": the object has been modified; please apply
your changes to the latest version and try again
E0130 15:32:06.477349 1 base_controller.go:279] "Unhandled Error" err="ConsoleOperator reconciliation failed: Operation cannot be fulfilled on deployments.apps \"console\": the object has been modified; please apply your changes
to the latest version and try again"
E0130 15:32:06.488808 1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.22.0-ec.1, 1 replicas available
I0130 15:32:06.553503 1 status_controller.go:230] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"DeploymentSyncDegraded: Operation cannot be fulfilled on deployments.
apps \"console\": the object has been modified; please apply your changes to the latest version and try again","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2026-01-30T15:32:06Z","message":"All is wel
l","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2026-01-30T08:14:45Z"
,"message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2026-01-27T16:53:37Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0130 15:32:06.822109 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Degraded message changed from "All is well" to "DeploymentSyncDegraded: Operation cannot be fulfilled on deployments.apps \"
console\": the object has been modified; please apply your changes to the latest version and try again",Progressing changed from True to False ("All is well")
# oc get cm console-config -n openshift-console -o yaml
apiVersion: v1
data:
console-config.yaml: |
apiVersion: console.openshift.io/v1
auth:
authType: openshift
clientID: console
clientSecretFile: /var/oauth-config/clientSecret
inactivityTimeoutSeconds: 3600
oauthEndpointCAFile: /var/oauth-serving-cert/ca-bundle.crt
clusterInfo:
consoleBaseAddress: https://console-openshift-console.apps.hubcluster-hp.nokia-stamp705.bos2.lab
controlPlaneTopology: HighlyAvailable
masterPublicURL: https://api.hubcluster-hp.nokia-stamp705.bos2.lab:6443
nodeArchitectures:
- amd64
nodeOperatingSystems:
- linux
releaseVersion: 4.22.0-ec.1
contentSecurityPolicyEnabled: true
customization:
branding: ocp
capabilities:
- name: LightspeedButton
visibility:
state: Enabled
- name: GettingStartedBanner
visibility:
state: Enabled
developerCatalog:
categories: null
types:
state: Enabled
documentationBaseURL: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.21/
perspectives:
- id: dev
visibility:
state: Disabled
i18nNamespaces:
- plugin__networking-console-plugin
- plugin__monitoring-plugin
- plugin__mce
- plugin__acm
- plugin__logging-view-plugin
kind: ConsoleConfig
monitoringInfo:
alertmanagerTenancyHost: alertmanager-main.openshift-monitoring.svc:9092
alertmanagerUserWorkloadHost: alertmanager-main.openshift-monitoring.svc:9094
plugins:
acm: https://console-chart-console-v2.open-cluster-management.svc.cluster.local:3000/plugin/
logging-view-plugin: https://logging.openshift-operators.svc.cluster.local:9443/
mce: https://console-mce-console.multicluster-engine.svc.cluster.local:3000/plugin/
monitoring-plugin: https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/
networking-console-plugin: https://networking-console-plugin.openshift-network-console.svc.cluster.local:9443/
odf-console: https://odf-console-service.openshift-storage.svc.cluster.local:9001/compatibility/
pluginsOrder:
- networking-console-plugin
- monitoring-plugin
- mce
- acm
- logging-view-plugin
- odf-console
providers: {}
proxy:
services:
- authorize: true
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/mce/console/
endpoint: https://console-mce-console.multicluster-engine.svc.cluster.local:3000
- authorize: true
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/acm/console/
endpoint: https://console-chart-console-v2.open-cluster-management.svc.cluster.local:3000
- authorize: true
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/logging-view-plugin/backend/
endpoint: https://logging-loki-gateway-http.openshift-logging.svc.cluster.local:8080
- authorize: true
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/logging-view-plugin/korrel8r/
endpoint: https://korrel8r.openshift-operators.svc.cluster.local:9443
- authorize: true
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/odf-console/provider-proxy/
endpoint: https://ux-backend-proxy.openshift-storage.svc.cluster.local:8888
- authorize: true
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/odf-console/rosa-prometheus/
endpoint: https://prometheus.openshift-storage.svc.cluster.local:9339
- authorize: false
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/odf-console/s3/
endpoint: https://s3.openshift-storage.svc.cluster.local:443
- authorize: false
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/odf-console/iam/
endpoint: https://iam.openshift-storage.svc.cluster.local:443
- authorize: false
caCertificate: ""
consoleAPIPath: /api/proxy/plugin/odf-console/internalRgwS3/
endpoint: https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:443
servingInfo:
bindAddress: https://[::]:8443
certFile: /var/serving-cert/tls.crt
keyFile: /var/serving-cert/tls.key
session: {}
telemetry:
CLUSTER_ID: 0f3a49c2-6538-4b02-bc6d-f1719e4394af
SEGMENT_API_HOST: console.redhat.com/connections/api/v1
SEGMENT_JS_HOST: console.redhat.com/connections/cdn
SEGMENT_PUBLIC_API_KEY: BnuS1RP39EmLQjP21ko67oDjhbl9zpNU
TELEMETER_CLIENT_DISABLED: "true"
kind: ConfigMap
metadata:
creationTimestamp: "2026-01-27T17:06:25Z"
labels:
app: console
name: console-config
namespace: openshift-console
ownerReferences:
- apiVersion: operator.openshift.io/v1
controller: true
kind: Console
name: cluster
uid: 955b03dc-d9f2-4dd2-97bb-2d6dcbc6d70f
resourceVersion: "5353479"
uid: fc9cecbb-65b9-4e94-8651-6c3ef7e0df91