I noticed the operator struggling to apply manifests right after the tech preview featureset is applied. I added some debugging logs and noticed this:

I0423 10:12:42.038474       1 render.go:99] rendering asset manifests/machine-os-puller/sa.yaml
I0423 10:12:42.052857       1 render.go:99] rendering asset manifests/machineconfigcontroller/kube-rbac-proxy-config.yaml
I0423 10:12:42.074064       1 render.go:99] rendering asset manifests/machineconfigcontroller/update-bootimages-validatingadmissionpolicy.yaml
I0423 10:12:42.100015       1 sync.go:791] Skipping retry in ApplyManifests for error: the server could not find the requested resource
I0423 10:12:42.120544       1 event.go:364] Event(v1.ObjectReference{Kind:"", Namespace:"openshift-machine-config-operator", Name:"machine-config", UID:"e90de8ef-1294-499a-94fd-ac870cb8fbcd", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'OperatorDegraded: MachineConfigControllerFailed' Failed to resync 4.16.0-0.nightly-2024-04-22-023835 because: failed to apply machine config controller manifests: the server could not find the requested resource

So it is most definitely failing on the VA, which is behind a k8s feature gate. Once the operator gets redeployed it resolves itself, which takes a while as the operator's node is queued last by design, so it'll be here until all other masters roll through. When the featureset is applied, the operator does restart, but it seems that is not enough. It needs to come up as a fresh pod after the previous one is terminated. Not sure why, but might be some setup we are missing with it being a kube native feature gate.

    4.16

    Always

    1. Bring up cluster with Default featureset
    2. Apply the TechPreview featureset
    3. Observe the operator pod logs