Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-57024

In OCL. Machine-os-builder pod does not release the lease when it is removed

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 5
    • Moderate
    • None
    • None
    • None
    • MCO Sprint 272, MCO Sprint 273
    • 2
    • Done
    • Release Note Not Required
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      The machine-os-builder pod is not releasing the leader lease when it is removed
      

      Version-Release number of selected component (if applicable):

      4.20.0-0.nightly-2025-06-02-135911
      

      How reproducible:

      Configure OCL and apply a MC, the machine-os-builder pod will be restarted and we will that it reports this error
      
      
      I0603 09:18:47.013288       1 leaderelection.go:257] attempting to acquire leader lease openshift-machine-config-operator/machine-os-builder...
      I0603 09:18:47.016444       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
      I0603 09:18:47.016458       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
      I0603 09:19:16.576468       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
      I0603 09:19:16.576485       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
      I0603 09:20:02.284367       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
      I0603 09:20:02.284383       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
      I0603 09:20:32.500332       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
      I0603 09:20:32.500345       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
      I0603 09:21:01.783799       1 leaderelection.go:456] lock is held by machine-os-builder-84bd7fcc4-85hkq_7e7b8645-7df0-4d47-a7ce-01ef0dda602d and has not yet expired
      I0603 09:21:01.783813       1 leaderelection.go:266] failed to acquire lease openshift-machine-config-operator/machine-os-builder
      I0603 09:21:45.008349       1 leaderelection.go:271] successfully acquired lease openshift-machine-config-operator/machine-os-builder
      
      It means that when the pod was removed it didnt release the lease. When the pod is removed, we can see these logs
      
      
      I0603 09:07:15.660983       1 cleaner.go:60] Cleaning up ephemeral objects from build "test-78001-8c09eda06d65b1ea60801f0cb1a0737b" using selector "machineconfiguration.openshift.io/ephemeral-build-object,machineconfiguration.openshift.io/machine-os-build=test-78001-8c09eda06d65b1ea60801f0cb1a0737b,machineconfiguration.openshift.io/on-cluster-layering,machineconfiguration.openshift.io/rendered-machine-config=rendered-worker-16d68ff4ee42b6aaffb0c9720260858a,machineconfiguration.openshift.io/target-machine-config-pool=worker"
      I0603 09:07:15.681872       1 helpers.go:111] Received shutdown signal: terminated. Delaying shutdown...
      I0603 09:07:15.725497       1 reconciler.go:976] Deleting image image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-worker-image:test-78001-8c09eda06d65b1ea60801f0cb1a0737b from internal registry for MachineOSBuild test-78001-8c09eda06d65b1ea60801f0cb1a0737b
      I0603 09:07:15.744335       1 reconciler.go:1108] Finished deleting MachineOSBuild "test-78001-8c09eda06d65b1ea60801f0cb1a0737b" after 91.61394ms
      I0603 09:07:20.726855       1 reconciler.go:1105] Updating MachineConfigPool "worker"
      I0603 09:07:20.726872       1 reconciler.go:1125] Syncing MachineOSConfigs and MachineOSBuilds
      I0603 09:07:20.726876       1 reconciler.go:1125] Syncing MachineOSConfigs
      I0603 09:07:20.726881       1 reconciler.go:1122] Finished syncing MachineOSConfigs after 5.252µs
      I0603 09:07:20.726885       1 reconciler.go:1125] Syncing MachineOSBuilds
      I0603 09:07:20.726888       1 reconciler.go:1122] Finished syncing MachineOSBuilds after 3.152µs
      I0603 09:07:20.726891       1 reconciler.go:1122] Finished syncing MachineOSConfigs and MachineOSBuilds after 19.702µs
      I0603 09:07:20.726894       1 reconciler.go:1108] Finished updating MachineConfigPool "worker" after 306.344µs
      I0603 09:07:25.754874       1 reconciler.go:1105] Updating MachineConfigPool "worker"
      I0603 09:07:25.754891       1 reconciler.go:1125] Syncing MachineOSConfigs and MachineOSBuilds
      I0603 09:07:25.754895       1 reconciler.go:1125] Syncing MachineOSConfigs
      I0603 09:07:25.754903       1 reconciler.go:1122] Finished syncing MachineOSConfigs after 7.083µs
      I0603 09:07:25.754911       1 reconciler.go:1125] Syncing MachineOSBuilds
      I0603 09:07:25.754914       1 reconciler.go:1122] Finished syncing MachineOSBuilds after 6.233µs
      I0603 09:07:25.754917       1 reconciler.go:1122] Finished syncing MachineOSConfigs and MachineOSBuilds after 28.045µs
      I0603 09:07:25.754922       1 reconciler.go:1108] Finished updating MachineConfigPool "worker" after 327.808µs
      
      We see in the log that the lease is not released
      
      

      Steps to Reproduce:

      Always
      

      Actual results:

      The machine-os-builder pod does not release the lease
      

      Expected results:

      The machine-os-builder pod should release the lease when it is shut down
      

      Additional info:

      
      Slack conversation: https://redhat-internal.slack.com/archives/GH7G2MANS/p1748594906387099
      

              zzlotnik@redhat.com Zack Zlotnik
              sregidor@redhat.com Sergio Regidor de la Rosa
              None
              None
              Sergio Regidor de la Rosa Sergio Regidor de la Rosa
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: