In clusters with OCB functionality enabled, sometimes the machine-os-builder pod is not restarted when we update the imageBuilderType.

What we have observed is that the pod is restarted if a build is running, but it is not restarted if we are not building anything.

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0-0.nightly-2023-09-12-195514   True        False         88m     Cluster version is 4.14.0-0.nightly-2023-09-12-195514

Always

1. Create the configuration resources needed by the OCB functionality.

To reproduce this issue we use an on-cluster-build-config configmap with an empty imageBuilderType

 oc patch cm/on-cluster-build-config -n openshift-machine-config-operator -p '{"data":{"imageBuilderType": ""}}'


2. Create a infra pool and label it so that it can use OCB functionality

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: ""


 oc label mcp/infra machineconfiguration.openshift.io/layering-enabled=

3. Wait for the build pod to finish.

4. Once the build has finished and it has been cleaned, update the imageBuilderType so that we use "custom-pod-builder" type now.

oc patch cm/on-cluster-build-config -n openshift-machine-config-operator -p '{"data":{"imageBuilderType": "custom-pod-builder"}}'

We waited for one hour, but the pod is never restarted.

$ oc get pods |grep build
machine-os-builder-6cfbd8d5d-xk6c5           1/1     Running   0          56m

$ oc logs machine-os-builder-6cfbd8d5d-xk6c5 |grep Type
I0914 08:40:23.910337       1 helpers.go:330] imageBuilderType empty, defaulting to "openshift-image-builder"

$ oc get cm on-cluster-build-config -o yaml |grep Type
  imageBuilderType: custom-pod-builder

When we update the imageBuilderType value, the machine-os-builder pod should be restarted.