Description of problem:

If a layered OS image (that is, an OS image with the appropriate MachineConfigs baked into it) is rolled out to a MachineConfigPool, one cannot easily revert to the original OS image.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Obtain the base OS container image for your cluster:

   $ oc get configmap/machine-config-osimageurl -o=jsonpath='{.data.baseOSContainerImage}'

1. Create a new MachineConfigPool called infra.
2. Obtain the current MachineConfig for the desired MachineConfigPool and store it on your local filesystem: 

   $ oc get machineconfig/mc-name -o json > mc-name.json

1. Create a Dockerfile like this:

FROM <base OS container image pullspec>
RUN mkdir -p /etc/machine-config-daemon
COPY ./mc-name.json /tmp/currentconfig
RUN mkdir -o /etc/machine-config-daemon && \
mv /tmp/currentconfig /etc/machine-config-daemon/currentconfig && \
exec -a ignition-apply /usr/lib/dracut/modules.d/30ignition/ignition --ignore-unsupported <(cat /etc/machine-config-daemon/currentconfig | jq '.spec.config') && \   ostree container commit

1. Build and push this image to a container registry:

$ podman build -t quay.io/org/repo:tag && podman push quay.io/org/repo:tag

1. Create a MachineConfig with the pullspec to your newly built image:

$ oc apply -f <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: infra
  name: os-image-override
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 3.2.0
    storage: {}
osImageURL: quay.io/org/repo:tag
EOF

1.  Label a worker node with the infra label:

$ oc label node/<nodename> l='noderole.kubernetes.io/infra='

1. Wait for the node to reboot into the newly built image and reconnect back to the cluster.
2. Remove the infra label from the node:

$ oc label node/<nodename> l='noderole.kubernetes.io/infra-'

1. Eventually, the node will become disconnected from the cluster and show NotReady when you run $ oc get nodes.

Actual results:

The node(s) in the target MachineConfigPool cannot revert back to the previously applied MachineConfig.

Expected results:

I would have expected the node(s) in the target MachineConfigPool to revert back to the previously applied MachineConfig.

Additional info:

• What's happening is the MCD first writes all the files to the node. It then writes the OS image. Writing the OS image sometimes overwrites the files / systemd units / et. al. written by the MCD, effectively reverting the nodes filesystem to that of the base OS image; modulo the parts of the filesystem that CoreOS allows one to mutate. Because of this, when the node reboots, none of the systemd units / configs are present, which causes the node to be unable to reconnect to the control plane.
• Interestingly, the aforementioned phenomenon does not occur in the TestOSImageURLOverride e2e test since the node is able to revert to its original config in that particular case. I suspect that since the custom OS image being deployed does not overwrite any of the files placed by the MCD, rpm-ostree leaves them alone when it applies the custom OS image. By comparison, in the aforementioned case, these files are effectively part of the disk image and are overwritten when rpm-ostree writes the original base OS image to the node.
• To confirm part of my hypothesis, I built a slightly modified version of the validateOnDiskState() function from the MCD into a separate binary (called machine-config-validator; code: https://github.com/cheesesashimi/machine-config-operator/commit/3bbbafda729efd9b23ff3294a2447cb95581bb27) and transferred it to my target node. Running this binary on a node whenever it is in this state shows that the expected on-disk state (obtained from the /etc/machine-config-daemon/currentconfig file) does not match the expected on-disk state. In particular, it shows numerous missing files and mismatched contents.
• A fairly simple fix could be to call validateOnDiskState() after writing the OS image, but before rebooting the node. If an error is detected, rewrite all of the files / systemd units / etc. before rebooting. This could also require some fairly advanced knowledge about rpm-ostree / ostree that I admittedly lack.
• To recover from this situation, one needs the following in place:
  • The cluster must be configured with at least one SSH key. This is because if the kubelet does not start, one cannot obtain a debug pod on the node.
  • If no external SSH access is available to the cluster, one can set up an SSH bastion pod with their SSH private key in a secret that they can $ oc exec into, then SSH into their node from there.
  • The following script, when run as sudo will retrieve the Machine Config Operator container, extract the machine-config-daemon binary from it, and emulate the first boot MachineConfig application process. This will write all of the expected files onto the node where needed and complete the node configuration, rebooting the node in the process.

#!/usr/bin/env bash

mco_image_pullspec="$1"
oc \
 image extract \
 --registry-config=/var/lib/kubelet/config.json \
 --confirm \
 "$mco_image_pullspec" \
 --path /usr/bin/machine-config-daemon:$HOME
cp /etc/machine-config-daemon/currentconfig /etc/ignition-machine-config-encapsulated.json
chmod +x "$HOME/machine-config-daemon"
"$HOME/machine-config-daemon" firstboot-complete-machineconfig