The machine-os-builder pod is not releasing the leader lease when it is removed

4.20.0-0.nightly-2025-06-02-135911

Configure OCL and apply a MC, the machine-os-builder pod will be restarted and we will that it reports this error


I0603 09:18:47.013288       1 leaderelection.go:257] attempting to acquire leader lease openshift-machine-config-operator/machine-os-builder...
I0603 09:18:47.016444       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
I0603 09:18:47.016458       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
I0603 09:19:16.576468       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
I0603 09:19:16.576485       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
I0603 09:20:02.284367       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
I0603 09:20:02.284383       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
I0603 09:20:32.500332       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
I0603 09:20:32.500345       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
I0603 09:21:01.783799       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
I0603 09:21:01.783813       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
I0603 09:21:45.008349       1 leaderelection.go:271] successfully acquired lease openshift-machine-config-operator/machine-os-builder

It means that when the pod was removed it didnt release the lease. When the pod is removed, we can see these logs


I0603 09:07:15.660983       1 cleaner.go:60] Cleaning up ephemeral objects from build "test-78001-8c09eda06d65b1ea60801f0cb1a0737b" using selector "machineconfiguration.openshift.io/ephemeral-build-object,machineconfiguration.openshift.io/machine-os-build=test-78001-8c09eda06d65b1ea60801f0cb1a0737b,machineconfiguration.openshift.io/on-cluster-layering,machineconfiguration.openshift.io/rendered-machine-config=rendered-worker-16d68ff4ee42b6aaffb0c9720260858a,machineconfiguration.openshift.io/target-machine-config-pool=worker"
I0603 09:07:15.681872       1 helpers.go:111] Received shutdown signal: terminated. Delaying shutdown...
I0603 09:07:15.725497       1 reconciler.go:976] Deleting image image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-worker-image:test-78001-8c09eda06d65b1ea60801f0cb1a0737b from internal registry for MachineOSBuild test-78001-8c09eda06d65b1ea60801f0cb1a0737b
I0603 09:07:15.744335       1 reconciler.go:1108] Finished deleting MachineOSBuild "test-78001-8c09eda06d65b1ea60801f0cb1a0737b" after 91.61394ms
I0603 09:07:20.726855       1 reconciler.go:1105] Updating MachineConfigPool "worker"
I0603 09:07:20.726872       1 reconciler.go:1125] Syncing MachineOSConfigs and MachineOSBuilds
I0603 09:07:20.726876       1 reconciler.go:1125] Syncing MachineOSConfigs
I0603 09:07:20.726881       1 reconciler.go:1122] Finished syncing MachineOSConfigs after 5.252µs
I0603 09:07:20.726885       1 reconciler.go:1125] Syncing MachineOSBuilds
I0603 09:07:20.726888       1 reconciler.go:1122] Finished syncing MachineOSBuilds after 3.152µs
I0603 09:07:20.726891       1 reconciler.go:1122] Finished syncing MachineOSConfigs and MachineOSBuilds after 19.702µs
I0603 09:07:20.726894       1 reconciler.go:1108] Finished updating MachineConfigPool "worker" after 306.344µs
I0603 09:07:25.754874       1 reconciler.go:1105] Updating MachineConfigPool "worker"
I0603 09:07:25.754891       1 reconciler.go:1125] Syncing MachineOSConfigs and MachineOSBuilds
I0603 09:07:25.754895       1 reconciler.go:1125] Syncing MachineOSConfigs
I0603 09:07:25.754903       1 reconciler.go:1122] Finished syncing MachineOSConfigs after 7.083µs
I0603 09:07:25.754911       1 reconciler.go:1125] Syncing MachineOSBuilds
I0603 09:07:25.754914       1 reconciler.go:1122] Finished syncing MachineOSBuilds after 6.233µs
I0603 09:07:25.754917       1 reconciler.go:1122] Finished syncing MachineOSConfigs and MachineOSBuilds after 28.045µs
I0603 09:07:25.754922       1 reconciler.go:1108] Finished updating MachineConfigPool "worker" after 327.808µs

We see in the log that the lease is not released

Always

The machine-os-builder pod does not release the lease

The machine-os-builder pod should release the lease when it is shut down

Slack conversation: https://redhat-internal.slack.com/archives/GH7G2MANS/p1748594906387099