After upgrading to 4.16.0 and then 4.16.1 on multiple clusters the console never seems to stop progressing and the openshift-console pods are almost constantly terminating and restarting.

Server Version: 4.16.1 

Seems always for me. This is happening on three Single Node OpenShift clusters I have upgraded to 4.16. I also observed it on a 4.16.0 AWS cluster that was upgraded before it was taken down.

    1. Upgrade a 4.15 cluster to (or install?) OpenShift 4.16 

The openshift-console never stops progressing and the pods are constantly terminating and restarting.

The pods should run without restarting every minute or so.

The console operator logs seem to be constantly looping:

E0710 14:42:47.267138       1 status.go:130] SyncLoopRefreshProgressing InProgress changes made during sync updates, additional sync expected
I0710 14:42:47.267426       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"2c4879d8-88f4-44b0-b6e4-fcc639606966", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeploymentUpdated' Updated Deployment.apps/console -n openshift-console because it changed
I0710 14:42:47.276911       1 helpers.go:201] Operator status changed:   &v1.OperatorStatus{
      ObservedGeneration: 2,
      Conditions: []v1.OperatorCondition{
          ... // 21 identical elements
          {Type: "DownloadsDeploymentSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:45 +0000 UTC"}},
          {Type: "SyncLoopRefreshDegraded", Status: "False", LastTransitionTime: {Time: s"2024-03-07 19:48:41 +0000 UTC"}},
-         {
-             Type:               "SyncLoopRefreshProgressing",
-             Status:             "False",
-             LastTransitionTime: s"2024-07-10 14:42:17 +0000 UTC",
-         },
+         {
+             Type:               "SyncLoopRefreshProgressing",
+             Status:             "True",
+             LastTransitionTime: s"2024-07-10 14:42:47.271690785 +0000 UTC m=+4414.624017206",
+             Reason:             "InProgress",
+             Message:            "changes made during sync updates, additional sync expected",
+         },
          {Type: "ConfigMapSyncDegraded", Status: "False", LastTransitionTime: {Time: s"2024-06-28 12:53:30 +0000 UTC"}},
          {Type: "ConfigMapSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:47 +0000 UTC"}},
          ... // 36 identical elements
      },
      Version:       "",
      ReadyReplicas: 1,
      Generations: []v1.GenerationStatus{
          {Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "downloads", ...},
          {
              ... // 2 identical fields
              Namespace:      "openshift-console",
              Name:           "console",
-             LastGeneration: 621,
+             LastGeneration: 622,
              Hash:           "",
          },
      },
  }
I0710 14:42:47.323761       1 status_controller.go:218] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2024-07-03T12:43:52Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2024-07-10T14:42:47Z","message":"SyncLoopRefreshProgressing: changes made during sync updates, additional sync expected","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitionTime":"2024-06-13T01:18:15Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2024-07-10T12:56:00Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2024-03-06T15:38:47Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0710 14:42:47.343866       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"2c4879d8-88f4-44b0-b6e4-fcc639606966", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from False to True ("SyncLoopRefreshProgressing: changes made during sync updates, additional sync expected")
I0710 14:42:48.423560       1 apps.go:154] Deployment "openshift-console/console" changes: {"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"strategy":{"rollingUpdate":{"maxSurge":null,"maxUnavailable":null}},"template":{"spec":{"containers":[{"command":["/opt/bridge/bin/bridge","--public-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:018642aee0af5ca28ccabd8f601922ef661eb5b61c457a690122b5e09f1caf46","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sleep","25"]}}},"livenessProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"name":"https","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory":"100Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name":"console-oauth-config","readOnly":true},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","name":"trusted-ca-bundle","readOnly":true},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"console-serving-cert"}},{"name":"console-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{"items":[{"key":"ca-bundle.crt","path":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
E0710 14:42:48.439728       1 status.go:130] SyncLoopRefreshProgressing InProgress changes made during sync updates, additional sync expected
I0710 14:42:48.440119       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"2c4879d8-88f4-44b0-b6e4-fcc639606966", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeploymentUpdated' Updated Deployment.apps/console -n openshift-console because it changed
E0710 14:42:50.040853       1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.16.1, 1 replicas available
I0710 14:42:50.049155       1 helpers.go:201] Operator status changed:   &v1.OperatorStatus{
      ObservedGeneration: 2,
      Conditions: []v1.OperatorCondition{
          ... // 21 identical elements
          {Type: "DownloadsDeploymentSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:45 +0000 UTC"}},
          {Type: "SyncLoopRefreshDegraded", Status: "False", LastTransitionTime: {Time: s"2024-03-07 19:48:41 +0000 UTC"}},
          {
              ... // 2 identical fields
              LastTransitionTime: {Time: s"2024-07-10 14:42:47 +0000 UTC"},
              Reason:             "InProgress",
              Message: strings.Join({
-                 "changes made during sync updates, additional sync expected",
+                 "working toward version 4.16.1, 1 replicas available",
              }, ""),
          },
          {Type: "ConfigMapSyncDegraded", Status: "False", LastTransitionTime: {Time: s"2024-06-28 12:53:30 +0000 UTC"}},
          {Type: "ConfigMapSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:47 +0000 UTC"}},
          ... // 36 identical elements
      },
      Version:       "",
      ReadyReplicas: 1,
      Generations:   {{Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "downloads", ...}, {Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "console", ...}},
  }
I0710 14:42:50.086000       1 status_controller.go:218] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2024-07-03T12:43:52Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2024-07-10T14:42:47Z","message":"SyncLoopRefreshProgressing: working toward version 4.16.1, 1 replicas available","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitionTime":"2024-06-13T01:18:15Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2024-07-10T12:56:00Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2024-03-06T15:38:47Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0710 14:42:50.107382       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"2c4879d8-88f4-44b0-b6e4-fcc639606966", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing message changed from "SyncLoopRefreshProgressing: changes made during sync updates, additional sync expected" to "SyncLoopRefreshProgressing: working toward version 4.16.1, 1 replicas available"
E0710 14:42:51.179366       1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.16.1, 1 replicas available
I0710 14:42:58.923936       1 helpers.go:201] Operator status changed:   &v1.OperatorStatus{
      ObservedGeneration: 2,
      Conditions: []v1.OperatorCondition{
          ... // 21 identical elements
          {Type: "DownloadsDeploymentSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:45 +0000 UTC"}},
          {Type: "SyncLoopRefreshDegraded", Status: "False", LastTransitionTime: {Time: s"2024-03-07 19:48:41 +0000 UTC"}},
-         {
-             Type:               "SyncLoopRefreshProgressing",
-             Status:             "True",
-             LastTransitionTime: s"2024-07-10 14:42:47 +0000 UTC",
-             Reason:             "InProgress",
-             Message:            "working toward version 4.16.1, 1 replicas available",
-         },
+         {
+             Type:               "SyncLoopRefreshProgressing",
+             Status:             "False",
+             LastTransitionTime: s"2024-07-10 14:42:58.917794386 +0000 UTC m=+4426.270120815",
+         },
          {Type: "ConfigMapSyncDegraded", Status: "False", LastTransitionTime: {Time: s"2024-06-28 12:53:30 +0000 UTC"}},
          {Type: "ConfigMapSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:47 +0000 UTC"}},
          ... // 36 identical elements
      },
      Version:       "",
      ReadyReplicas: 1,
      Generations:   {{Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "downloads", ...}, {Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "console", ...}},
  }
I0710 14:42:58.957379       1 status_controller.go:218] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2024-07-03T12:43:52Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2024-07-10T14:42:58Z","message":"All is well","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2024-06-13T01:18:15Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2024-07-10T12:56:00Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2024-03-06T15:38:47Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0710 14:42:58.972697       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"2c4879d8-88f4-44b0-b6e4-fcc639606966", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from True to False ("All is well")
E0710 14:42:59.061618       1 sync_v400.go:468] telemetry config error: HTTP request failed with status '429 Too Many Requests'
I0710 14:42:59.082742       1 core.go:341] ConfigMap "openshift-console/console-config" changes: {"apiVersion":"v1","data":{"console-config.yaml":"apiVersion: console.openshift.io/v1\nauth:\n  authType: openshift\n  clientID: console\n  clientSecretFile: /var/oauth-config/clientSecret\n  oauthEndpointCAFile: /var/oauth-serving-cert/ca-bundle.crt\nclusterInfo:\n  consoleBaseAddress: https://console-openshift-console.apps.openshift4-ppc.montleon.net\n  controlPlaneTopology: SingleReplica\n  masterPublicURL: https://api.openshift4-ppc.montleon.net:6443\n  nodeArchitectures:\n  - ppc64le\n  nodeOperatingSystems:\n  - linux\n  releaseVersion: 4.16.1\ncustomization:\n  branding: ocp\n  documentationBaseURL: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.16/\nkind: ConsoleConfig\nmonitoringInfo:\n  alertmanagerTenancyHost: alertmanager-main.openshift-monitoring.svc:9092\n  alertmanagerUserWorkloadHost: alertmanager-main.openshift-monitoring.svc:9094\nplugins:\n  monitoring-plugin: https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/\nproviders: {}\nservingInfo:\n  bindAddress: https://[::]:8443\n  certFile: /var/serving-cert/tls.crt\n  keyFile: /var/serving-cert/tls.key\nsession: {}\ntelemetry:\n  CLUSTER_ID: 72854345-647a-434d-b4e1-6396f5d66c79\n  ORGANIZATION_ID: \"\"\n  SEGMENT_API_HOST: console.redhat.com/connections/api/v1\n  SEGMENT_JS_HOST: console.redhat.com/connections/cdn\n  SEGMENT_PUBLIC_API_KEY: BnuS1RP39EmLQjP21ko67oDjhbl9zpNU\n"},"kind":"ConfigMap","metadata":{"creationTimestamp":null,"managedFields":null,"resourceVersion":null,"uid":null}}
I0710 14:42:59.083208       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"2c4879d8-88f4-44b0-b6e4-fcc639606966", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/console-config -n openshift-console:
cause by changes in data.console-config.yaml
I0710 14:42:59.099409       1 apps.go:154] Deployment "openshift-console/console" changes: {"metadata":{"annotations":{"console.openshift.io/console-config-version":"1651506","operator.openshift.io/spec-hash":"ae6bba05f6db1862602c621087974548a636dbe5e4ea07ff39f2d68dd8b743d1"}},"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"strategy":{"rollingUpdate":{"maxSurge":null,"maxUnavailable":null}},"template":{"metadata":{"annotations":{"console.openshift.io/console-config-version":"1651506"}},"spec":{"containers":[{"command":["/opt/bridge/bin/bridge","--public-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:018642aee0af5ca28ccabd8f601922ef661eb5b61c457a690122b5e09f1caf46","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sleep","25"]}}},"livenessProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"name":"https","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory":"100Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name":"console-oauth-config","readOnly":true},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","name":"trusted-ca-bundle","readOnly":true},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"console-serving-cert"}},{"name":"console-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{"items":[{"key":"ca-bundle.crt","path":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
E0710 14:42:59.121761       1 status.go:130] SyncLoopRefreshProgressing InProgress changes made during sync updates, additional sync expected
I0710 14:42:59.122146       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"2c4879d8-88f4-44b0-b6e4-fcc639606966", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeploymentUpdated' Updated Deployment.apps/console -n openshift-console because it changed
I0710 14:42:59.132751       1 helpers.go:201] Operator status changed:   &v1.OperatorStatus{
      ObservedGeneration: 2,
      Conditions: []v1.OperatorCondition{
          ... // 21 identical elements
          {Type: "DownloadsDeploymentSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:45 +0000 UTC"}},
          {Type: "SyncLoopRefreshDegraded", Status: "False", LastTransitionTime: {Time: s"2024-03-07 19:48:41 +0000 UTC"}},
-         {
-             Type:               "SyncLoopRefreshProgressing",
-             Status:             "False",
-             LastTransitionTime: s"2024-07-10 14:42:58 +0000 UTC",
-         },
+         {
+             Type:               "SyncLoopRefreshProgressing",
+             Status:             "True",
+             LastTransitionTime: s"2024-07-10 14:42:59.127589979 +0000 UTC m=+4426.479916439",
+             Reason:             "InProgress",
+             Message:            "changes made during sync updates, additional sync expected",
+         },
          {Type: "ConfigMapSyncDegraded", Status: "False", LastTransitionTime: {Time: s"2024-06-28 12:53:30 +0000 UTC"}},
          {Type: "ConfigMapSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-02-09 23:46:47 +0000 UTC"}},
          ... // 36 identical elements
      },
      Version:       "",
      ReadyReplicas: 1,
      Generations: []v1.GenerationStatus{
          {Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "downloads", ...},
          {
              ... // 2 identical fields
              Namespace:      "openshift-console",
              Name:           "console",
-             LastGeneration: 622,
+             LastGeneration: 623,
              Hash:           "",
          },
      },
  }