-
Bug
-
Resolution: Done
-
Blocker
-
None
-
None
-
Incidents & Support
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Starting with 4.20.0-0.nightly-2025-07-09-121230 which is the first payload installing 4.19.0-0.nightly-2025-07-08-172809 for minor upgrades we see 4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade failures due to pathological events like
event happened 25 times, something is wrong: namespace/openshift-kube-apiserver hmsg/cf99229525 - reason/SecretUpdated Updated Secret/kube-controller-manager-client-cert-key -n openshift-config-managed because it changed (05:57:30Z) result=reject event happened 28 times, something is wrong: namespace/openshift-kube-apiserver hmsg/d7474cfe31 - reason/SecretUpdated Updated Secret/kube-scheduler-client-cert-key -n openshift-config-managed because it changed (05:57:05Z) result=reject event happened 28 times, something is wrong: namespace/openshift-kube-apiserver hmsg/e3d56f78d6 - reason/SecretUpdated Updated Secret/control-plane-node-admin-client-cert-key -n openshift-kube-apiserver because it changed (05:54:13Z) result=reject event happened 34 times, something is wrong: namespace/openshift-kube-apiserver hmsg/7727920ddf - reason/SecretUpdated Updated Secret/check-endpoints-client-cert-key -n openshift-kube-apiserver because it changed (05:57:27Z) result=reject event happened 61 times, something is wrong: namespace/openshift-kube-apiserver hmsg/819d5019a3 - reason/ConfigMapUpdated Updated ConfigMap/node-system-admin-ca -n openshift-kube-apiserver-operator because it changed (05:54:17Z) result=reject event happened 53 times, something is wrong: namespace/openshift-kube-apiserver hmsg/52ea701932 - reason/ConfigMapUpdated Updated ConfigMap/kube-apiserver-aggregator-client-ca -n openshift-config-managed because it changed (05:54:16Z) result=reject event happened 42 times, something is wrong: namespace/openshift-kube-apiserver hmsg/1bec32f94c - reason/ConfigMapUpdated Updated ConfigMap/kube-control-plane-signer-ca -n openshift-kube-apiserver-operator because it changed (05:57:29Z) result=reject }
and
event happened 92 times, something is wrong: namespace/openshift-kube-scheduler-operator deployment/openshift-kube-scheduler-operator hmsg/6a547895ad - reason/OperatorStatusChanged Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready\nResourceSyncControllerDegraded: Operation cannot be fulfilled on secrets \"kube-scheduler-client-cert-key\": the object has been modified; please apply your changes to the latest version and try again" (05:57:43Z) result=reject event happened 92 times, something is wrong: namespace/openshift-kube-scheduler-operator deployment/openshift-kube-scheduler-operator hmsg/d3166e73d5 - reason/OperatorStatusChanged Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nResourceSyncControllerDegraded: Operation cannot be fulfilled on secrets \"kube-scheduler-client-cert-key\": the object has been modified; please apply your changes to the latest version and try again" to "NodeControllerDegraded: All master nodes are ready" (05:57:43Z) result=reject event happened 79 times, something is wrong: namespace/openshift-kube-scheduler-operator deployment/openshift-kube-scheduler-operator hmsg/64a3018a4d - reason/SecretUpdateFailed Failed to update Secret/kube-scheduler-client-cert-key -n openshift-kube-scheduler: Operation cannot be fulfilled on secrets "kube-scheduler-client-cert-key": the object has been modified; please apply your changes to the latest version and try again (05:57:43Z) result=reject }
etc.
Events shows entries like
05:57:07 (x35) openshift-kube-apiserver-operator kube-apiserver-operator-cert-rotation-controller kube-apiserver-operator SecretUpdated Updated Secret/control-plane-node-admin-client-cert-key -n openshift-kube-apiserver because it changed
Which is around the time kube-apiserver clusteroperator is being upgraded
- links to