After update to OpenShift Container Platform 4.13.4 the MachineConfigPool is found in degraded state because the MachineConfigDaemon is failing to complete the upgrade and is actually stuck in a loop (post OpenShift Container Platform 4 - Node reboot).

2023-07-04T08:31:29.132976399Z I0704 08:31:29.132963    2820 update.go:1342] Removed stale file "/etc/sysctl.d/enable-userfaultfd.conf"
2023-07-04T08:31:29.133011208Z I0704 08:31:29.133002    2820 update.go:2055] Removing SIGTERM protection
2023-07-04T08:31:29.133025668Z E0704 08:31:29.133014    2820 writer.go:200] Marking Degraded due to: exit status 1
2023-07-04T08:32:29.146841055Z I0704 08:32:29.146781    2820 daemon.go:582] State and Reason: Degraded exit status 1
2023-07-04T08:32:30.761725462Z I0704 08:32:30.761681    2820 rpm-ostree.go:382] Running captured: rpm-ostree kargs
2023-07-04T08:32:30.870536786Z I0704 08:32:30.870487    2820 daemon.go:666] Preflight config drift check successful (took 1.72147513s)
2023-07-04T08:32:30.870536786Z I0704 08:32:30.870516    2820 update.go:2047] Adding SIGTERM protection
2023-07-04T08:32:30.917766296Z I0704 08:32:30.917725    2820 update.go:500] Checking Reconcilable for config rendered-foo-worker-infra-8043a0ad5ba74d790e9406980e180bc2 to rendered-foo-worker-infra-dd1d2c328b322572f3eefd2e547ba8d4
2023-07-04T08:32:31.015552082Z I0704 08:32:31.015511    2820 update.go:2025] Starting update from rendered-foo-worker-infra-8043a0ad5ba74d790e9406980e180bc2 to rendered-foo-worker-infra-dd1d2c328b322572f3eefd2e547ba8d4: &{osUpdate:true kargs:false fips:false passwd:false files:true units:true kernelType:false extensions:false}
2023-07-04T08:32:31.017042947Z I0704 08:32:31.017012    2820 helpers.go:996] File diff: /etc/sysctl.d/enable-userfaultfd.conf was added
2023-07-04T08:32:31.017042947Z I0704 08:32:31.017039    2820 helpers.go:1000] File diff: detected change to /etc/kubernetes/kubelet-ca.crt
2023-07-04T08:32:31.017062848Z I0704 08:32:31.017052    2820 helpers.go:1000] File diff: detected change to /etc/crio/crio.conf.d/00-default
2023-07-04T08:32:31.017095290Z I0704 08:32:31.017084    2820 update.go:2025] drain is already completed on this node
2023-07-04T08:32:31.018306989Z I0704 08:32:31.018293    2820 update.go:1213] Updating files
2023-07-04T08:32:31.018306989Z I0704 08:32:31.018301    2820 file_writers.go:163] Writing file "/usr/local/bin/nm-clean-initrd-state.sh"
2023-07-04T08:32:31.021545953Z I0704 08:32:31.021522    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/01-ipv6.conf"
2023-07-04T08:32:31.023929555Z I0704 08:32:31.023911    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/20-keyfiles.conf"
2023-07-04T08:32:31.026312448Z I0704 08:32:31.026290    2820 file_writers.go:163] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
2023-07-04T08:32:31.028802610Z I0704 08:32:31.028784    2820 file_writers.go:163] Writing file "/etc/audit/rules.d/mco-audit-quiet-containers.rules"
2023-07-04T08:32:31.031142225Z I0704 08:32:31.031126    2820 file_writers.go:163] Writing file "/etc/tmpfiles.d/cleanup-cni.conf"
2023-07-04T08:32:31.033510457Z I0704 08:32:31.033486    2820 file_writers.go:163] Writing file "/etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem"
2023-07-04T08:32:31.035047768Z I0704 08:32:31.035033    2820 file_writers.go:163] Writing file "/usr/local/bin/configure-ovs.sh"
2023-07-04T08:32:31.038911142Z I0704 08:32:31.038887    2820 file_writers.go:163] Writing file "/etc/containers/storage.conf"
2023-07-04T08:32:31.041281815Z I0704 08:32:31.041262    2820 file_writers.go:163] Writing file "/etc/mco/proxy.env"
2023-07-04T08:32:31.043616520Z I0704 08:32:31.043596    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/10-default-env-godebug.conf"
2023-07-04T08:32:31.046045132Z I0704 08:32:31.046026    2820 file_writers.go:163] Writing file "/etc/modules-load.d/iptables.conf"
2023-07-04T08:32:31.048663171Z I0704 08:32:31.048643    2820 file_writers.go:163] Writing file "/etc/node-sizing-enabled.env"
2023-07-04T08:32:31.051454995Z I0704 08:32:31.051433    2820 file_writers.go:163] Writing file "/usr/local/sbin/dynamic-system-reserved-calc.sh"
2023-07-04T08:32:31.054159138Z I0704 08:32:31.054139    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/kubelet-cgroups.conf"
2023-07-04T08:32:31.056602335Z I0704 08:32:31.056582    2820 file_writers.go:163] Writing file "/etc/systemd/system/kubelet.service.d/20-logging.conf"
2023-07-04T08:32:31.060440920Z I0704 08:32:31.060417    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/sdn.conf"
2023-07-04T08:32:31.063127248Z I0704 08:32:31.063102    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh"
2023-07-04T08:32:31.066955881Z I0704 08:32:31.066930    2820 file_writers.go:163] Writing file "/var/lib/kubelet/config.json"
2023-07-04T08:32:31.069906347Z I0704 08:32:31.069886    2820 file_writers.go:163] Writing file "/etc/kubernetes/ca.crt"
2023-07-04T08:32:31.072897180Z I0704 08:32:31.072880    2820 file_writers.go:163] Writing file "/etc/sysctl.d/arp.conf"
2023-07-04T08:32:31.075549695Z I0704 08:32:31.075533    2820 file_writers.go:163] Writing file "/etc/sysctl.d/forward.conf"
2023-07-04T08:32:31.078204914Z I0704 08:32:31.078186    2820 file_writers.go:163] Writing file "/etc/sysctl.d/inotify.conf"
2023-07-04T08:32:31.080799835Z I0704 08:32:31.080779    2820 file_writers.go:163] Writing file "/etc/sysctl.d/enable-userfaultfd.conf"
2023-07-04T08:32:31.085621738Z I0704 08:32:31.085591    2820 file_writers.go:163] Writing file "/etc/sysctl.d/vm-max-map.conf"
2023-07-04T08:32:31.088312397Z I0704 08:32:31.088292    2820 file_writers.go:163] Writing file "/usr/local/bin/mco-hostname"
2023-07-04T08:32:31.090875202Z I0704 08:32:31.090857    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet-plugins/volume/exec/.dummy"
2023-07-04T08:32:31.092065448Z I0704 08:32:31.092041    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
2023-07-04T08:32:31.094580986Z I0704 08:32:31.094561    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf"
2023-07-04T08:32:31.097009942Z I0704 08:32:31.096987    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/00-default"
2023-07-04T08:32:31.099577945Z I0704 08:32:31.099557    2820 file_writers.go:163] Writing file "/etc/containers/policy.json"
2023-07-04T08:32:31.101999510Z I0704 08:32:31.101982    2820 file_writers.go:163] Writing file "/etc/kubernetes/cloud.conf"
2023-07-04T08:32:31.104461465Z I0704 08:32:31.104442    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet.conf"
2023-07-04T08:32:31.107166866Z I0704 08:32:31.107149    2820 file_writers.go:163] Writing file "/usr/local/bin/kubenswrapper"
2023-07-04T08:32:31.109576732Z I0704 08:32:31.109556    2820 file_writers.go:163] Writing file "/etc/chrony.conf"
2023-07-04T08:32:31.112003946Z I0704 08:32:31.111984    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/01-mc-addInheritableCapabilities"
2023-07-04T08:32:31.114783721Z I0704 08:32:31.114759    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf.d/01-image-searchRegistries.conf"
2023-07-04T08:32:31.117886388Z I0704 08:32:31.117866    2820 file_writers.go:223] Writing systemd unit "NetworkManager-clean-initrd-state.service"
2023-07-04T08:32:31.120337881Z I0704 08:32:31.120321    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:31.123435190Z I0704 08:32:31.123415    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.123435190Z I0704 08:32:31.123430    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-profile-unix-socket.conf"
2023-07-04T08:32:31.125857113Z I0704 08:32:31.125832    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:31.390486704Z I0704 08:32:31.390445    2820 update.go:1483] Preset systemd unit crio.service
2023-07-04T08:32:31.390486704Z I0704 08:32:31.390472    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:31.398987268Z I0704 08:32:31.398945    2820 update.go:1520] Could not reset unit preset for docker.socket, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file docker.socket does not exist.
2023-07-04T08:32:31.398987268Z )
2023-07-04T08:32:31.398987268Z I0704 08:32:31.398960    2820 file_writers.go:223] Writing systemd unit "kubelet-auto-node-size.service"
2023-07-04T08:32:31.401587733Z I0704 08:32:31.401571    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:31.404448091Z I0704 08:32:31.404434    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.404448091Z I0704 08:32:31.404443    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:31.406920375Z I0704 08:32:31.406906    2820 file_writers.go:223] Writing systemd unit "kubelet.service"
2023-07-04T08:32:31.409311616Z I0704 08:32:31.409298    2820 file_writers.go:223] Writing systemd unit "kubens.service"
2023-07-04T08:32:31.411446522Z I0704 08:32:31.411433    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-firstboot.service"
2023-07-04T08:32:31.413867275Z I0704 08:32:31.413833    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-pull.service"
2023-07-04T08:32:31.416563042Z I0704 08:32:31.416531    2820 file_writers.go:223] Writing systemd unit "node-valid-hostname.service"
2023-07-04T08:32:31.419057272Z I0704 08:32:31.419028    2820 file_writers.go:223] Writing systemd unit "nodeip-configuration.service"
2023-07-04T08:32:31.421387560Z I0704 08:32:31.421363    2820 file_writers.go:223] Writing systemd unit "ovs-configuration.service"
2023-07-04T08:32:31.423804731Z I0704 08:32:31.423785    2820 file_writers.go:137] Writing systemd unit dropin "10-ovs-vswitchd-restart.conf"
2023-07-04T08:32:31.697822673Z I0704 08:32:31.697776    2820 update.go:1483] Preset systemd unit ovs-vswitchd.service
2023-07-04T08:32:31.697888155Z I0704 08:32:31.697872    2820 file_writers.go:137] Writing systemd unit dropin "10-ovsdb-restart.conf"
2023-07-04T08:32:31.701610253Z I0704 08:32:31.701579    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.706806366Z I0704 08:32:31.706775    2820 update.go:1520] Could not reset unit preset for pivot.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file pivot.service does not exist.
2023-07-04T08:32:31.706806366Z )
2023-07-04T08:32:31.706806366Z I0704 08:32:31.706793    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.976103084Z I0704 08:32:31.976063    2820 update.go:1483] Preset systemd unit rpm-ostreed.service
2023-07-04T08:32:31.976103084Z I0704 08:32:31.976085    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:31.984458031Z I0704 08:32:31.984421    2820 update.go:1520] Could not reset unit preset for zincati.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file zincati.service does not exist.
2023-07-04T08:32:31.984458031Z )
2023-07-04T08:32:32.248022810Z I0704 08:32:32.247979    2820 update.go:1461] Enabled systemd units: [NetworkManager-clean-initrd-state.service kubelet-auto-node-size.service kubelet.service machine-config-daemon-firstboot.service machine-config-daemon-pull.service node-valid-hostname.service openvswitch.service ovs-configuration.service ovsdb-server.service]
2023-07-04T08:32:32.521333265Z I0704 08:32:32.521285    2820 update.go:1472] Disabled systemd units [kubens.service nodeip-configuration.service]
2023-07-04T08:32:32.521333265Z I0704 08:32:32.521312    2820 update.go:1279] Deleting stale data
2023-07-04T08:32:32.521443531Z I0704 08:32:32.521430    2820 update.go:1580] Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
2023-07-04T08:32:32.521443531Z I0704 08:32:32.521440    2820 update.go:1547] Creating missing SSH key dir at /home/core/.ssh/authorized_keys.d
2023-07-04T08:32:32.529818660Z I0704 08:32:32.529792    2820 update.go:1213] Updating files
2023-07-04T08:32:32.529818660Z I0704 08:32:32.529805    2820 file_writers.go:163] Writing file "/usr/local/bin/nm-clean-initrd-state.sh"
2023-07-04T08:32:32.532664744Z I0704 08:32:32.532642    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/01-ipv6.conf"
2023-07-04T08:32:32.535250673Z I0704 08:32:32.535226    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/20-keyfiles.conf"
2023-07-04T08:32:32.537906978Z I0704 08:32:32.537885    2820 file_writers.go:163] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
2023-07-04T08:32:32.540446432Z I0704 08:32:32.540426    2820 file_writers.go:163] Writing file "/etc/audit/rules.d/mco-audit-quiet-containers.rules"
2023-07-04T08:32:32.543767171Z I0704 08:32:32.543744    2820 file_writers.go:163] Writing file "/etc/tmpfiles.d/cleanup-cni.conf"
2023-07-04T08:32:32.546217109Z I0704 08:32:32.546200    2820 file_writers.go:163] Writing file "/etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem"
2023-07-04T08:32:32.547950869Z I0704 08:32:32.547936    2820 file_writers.go:163] Writing file "/usr/local/bin/configure-ovs.sh"
2023-07-04T08:32:32.552111397Z I0704 08:32:32.552090    2820 file_writers.go:163] Writing file "/etc/containers/storage.conf"
2023-07-04T08:32:32.554524136Z I0704 08:32:32.554507    2820 file_writers.go:163] Writing file "/etc/mco/proxy.env"
2023-07-04T08:32:32.557275961Z I0704 08:32:32.557248    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/10-default-env-godebug.conf"
2023-07-04T08:32:32.559600586Z I0704 08:32:32.559582    2820 file_writers.go:163] Writing file "/etc/modules-load.d/iptables.conf"
2023-07-04T08:32:32.562101524Z I0704 08:32:32.562064    2820 file_writers.go:163] Writing file "/etc/node-sizing-enabled.env"
2023-07-04T08:32:32.564682417Z I0704 08:32:32.564661    2820 file_writers.go:163] Writing file "/usr/local/sbin/dynamic-system-reserved-calc.sh"
2023-07-04T08:32:32.567178788Z I0704 08:32:32.567159    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/kubelet-cgroups.conf"
2023-07-04T08:32:32.569635827Z I0704 08:32:32.569612    2820 file_writers.go:163] Writing file "/etc/systemd/system/kubelet.service.d/20-logging.conf"
2023-07-04T08:32:32.572151315Z I0704 08:32:32.572130    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/sdn.conf"
2023-07-04T08:32:32.574782788Z I0704 08:32:32.574709    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh"
2023-07-04T08:32:32.577763285Z I0704 08:32:32.577741    2820 file_writers.go:163] Writing file "/var/lib/kubelet/config.json"
2023-07-04T08:32:32.580253818Z I0704 08:32:32.580237    2820 file_writers.go:163] Writing file "/etc/kubernetes/ca.crt"
2023-07-04T08:32:32.582696360Z I0704 08:32:32.582681    2820 file_writers.go:163] Writing file "/etc/sysctl.d/arp.conf"
2023-07-04T08:32:32.585085435Z I0704 08:32:32.585068    2820 file_writers.go:163] Writing file "/etc/sysctl.d/forward.conf"
2023-07-04T08:32:32.588642085Z I0704 08:32:32.588624    2820 file_writers.go:163] Writing file "/etc/sysctl.d/inotify.conf"
2023-07-04T08:32:32.594399117Z I0704 08:32:32.594381    2820 file_writers.go:163] Writing file "/etc/sysctl.d/vm-max-map.conf"
2023-07-04T08:32:32.597034484Z I0704 08:32:32.597013    2820 file_writers.go:163] Writing file "/usr/local/bin/mco-hostname"
2023-07-04T08:32:32.599620615Z I0704 08:32:32.599602    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet-plugins/volume/exec/.dummy"
2023-07-04T08:32:32.600955688Z I0704 08:32:32.600919    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
2023-07-04T08:32:32.603456084Z I0704 08:32:32.603439    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf"
2023-07-04T08:32:32.606369537Z I0704 08:32:32.606350    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/00-default"
2023-07-04T08:32:32.610556300Z I0704 08:32:32.610539    2820 file_writers.go:163] Writing file "/etc/containers/policy.json"
2023-07-04T08:32:32.613126991Z I0704 08:32:32.613110    2820 file_writers.go:163] Writing file "/etc/kubernetes/cloud.conf"
2023-07-04T08:32:32.615683508Z I0704 08:32:32.615630    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet.conf"
2023-07-04T08:32:32.618887956Z I0704 08:32:32.618837    2820 file_writers.go:163] Writing file "/usr/local/bin/kubenswrapper"
2023-07-04T08:32:32.621550528Z I0704 08:32:32.621534    2820 file_writers.go:163] Writing file "/etc/chrony.conf"
2023-07-04T08:32:32.624263775Z I0704 08:32:32.624245    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/01-mc-addInheritableCapabilities"
2023-07-04T08:32:32.627698830Z I0704 08:32:32.627642    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf.d/01-image-searchRegistries.conf"
2023-07-04T08:32:32.630200731Z I0704 08:32:32.630184    2820 file_writers.go:223] Writing systemd unit "NetworkManager-clean-initrd-state.service"
2023-07-04T08:32:32.632449548Z I0704 08:32:32.632433    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:32.635029500Z I0704 08:32:32.635012    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:32.635029500Z I0704 08:32:32.635022    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-profile-unix-socket.conf"
2023-07-04T08:32:32.637234367Z I0704 08:32:32.637180    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:32.904509018Z I0704 08:32:32.904452    2820 update.go:1483] Preset systemd unit crio.service
2023-07-04T08:32:32.904509018Z I0704 08:32:32.904473    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:32.913062652Z I0704 08:32:32.913026    2820 update.go:1520] Could not reset unit preset for docker.socket, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file docker.socket does not exist.
2023-07-04T08:32:32.913062652Z )
2023-07-04T08:32:32.913140214Z I0704 08:32:32.913110    2820 file_writers.go:223] Writing systemd unit "kubelet-auto-node-size.service"
2023-07-04T08:32:32.915969332Z I0704 08:32:32.915952    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:32.919073945Z I0704 08:32:32.919051    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:32.919073945Z I0704 08:32:32.919061    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:32.922111393Z I0704 08:32:32.922093    2820 file_writers.go:223] Writing systemd unit "kubelet.service"
2023-07-04T08:32:32.925123428Z I0704 08:32:32.925099    2820 file_writers.go:223] Writing systemd unit "kubens.service"
2023-07-04T08:32:32.927479322Z I0704 08:32:32.927461    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-firstboot.service"
2023-07-04T08:32:32.931479040Z I0704 08:32:32.931460    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-pull.service"
2023-07-04T08:32:32.933850616Z I0704 08:32:32.933834    2820 file_writers.go:223] Writing systemd unit "node-valid-hostname.service"
2023-07-04T08:32:32.936694241Z I0704 08:32:32.936674    2820 file_writers.go:223] Writing systemd unit "nodeip-configuration.service"
2023-07-04T08:32:32.939181286Z I0704 08:32:32.939165    2820 file_writers.go:223] Writing systemd unit "ovs-configuration.service"
2023-07-04T08:32:32.941906819Z I0704 08:32:32.941886    2820 file_writers.go:137] Writing systemd unit dropin "10-ovs-vswitchd-restart.conf"
2023-07-04T08:32:33.208984072Z I0704 08:32:33.208942    2820 update.go:1483] Preset systemd unit ovs-vswitchd.service
2023-07-04T08:32:33.208984072Z I0704 08:32:33.208961    2820 file_writers.go:137] Writing systemd unit dropin "10-ovsdb-restart.conf"
2023-07-04T08:32:33.212118544Z I0704 08:32:33.212091    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:33.217372730Z I0704 08:32:33.217342    2820 update.go:1520] Could not reset unit preset for pivot.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file pivot.service does not exist.
2023-07-04T08:32:33.217372730Z )
2023-07-04T08:32:33.217372730Z I0704 08:32:33.217355    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:33.478951518Z I0704 08:32:33.478908    2820 update.go:1483] Preset systemd unit rpm-ostreed.service
2023-07-04T08:32:33.478951518Z I0704 08:32:33.478931    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:33.487605830Z I0704 08:32:33.487563    2820 update.go:1520] Could not reset unit preset for zincati.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file zincati.service does not exist.
2023-07-04T08:32:33.487605830Z )
2023-07-04T08:32:33.753396023Z I0704 08:32:33.753350    2820 update.go:1461] Enabled systemd units: [NetworkManager-clean-initrd-state.service kubelet-auto-node-size.service kubelet.service machine-config-daemon-firstboot.service machine-config-daemon-pull.service node-valid-hostname.service openvswitch.service ovs-configuration.service ovsdb-server.service]
2023-07-04T08:32:34.016390168Z I0704 08:32:34.016338    2820 update.go:1472] Disabled systemd units [kubens.service nodeip-configuration.service]
2023-07-04T08:32:34.016390168Z I0704 08:32:34.016367    2820 update.go:1279] Deleting stale data
2023-07-04T08:32:34.016470522Z I0704 08:32:34.016456    2820 update.go:1342] Removed stale file "/etc/sysctl.d/enable-userfaultfd.conf"
2023-07-04T08:32:34.016514286Z I0704 08:32:34.016496    2820 update.go:2055] Removing SIGTERM protection
2023-07-04T08:32:34.016523204Z E0704 08:32:34.016517    2820 writer.go:200] Marking Degraded due to: exit status 1
2023-07-04T08:33:34.036229992Z I0704 08:33:34.036176    2820 daemon.go:582] State and Reason: Degraded exit status 1

Above is the loop detailed, found in the respective machine-config-daemon pod. It basically works as the upgrade is properly triggered and running but once the system is come up with the latest version, the machine-config-daemon is stuck in the given loop.

When deleting the problematic Machine, a new Machine is being created and successfully able to join the OpenShift Container Platform 4 - Cluster without similar issues reported in machine-config-daemon.

OpenShift Container Platform 4.13.4

N/A

1. N/A - only see on one specific OpenShift Container Platform 4.13.4 - Cluster

$ omc get mcp foo-worker-infra -o json | jq '.status.conditions'
[
  {
    "lastTransitionTime": "2023-02-20T14:06:09Z",
    "message": "",
    "reason": "",
    "status": "False",
    "type": "RenderDegraded"
  },
  {
    "lastTransitionTime": "2023-06-23T05:08:08Z",
    "message": "",
    "reason": "",
    "status": "False",
    "type": "Updated"
  },
  {
    "lastTransitionTime": "2023-06-23T05:08:08Z",
    "message": "All nodes are updating to rendered-foo-worker-infra-dd1d2c328b322572f3eefd2e547ba8d4",
    "reason": "",
    "status": "True",
    "type": "Updating"
  },
  {
    "lastTransitionTime": "2023-07-04T08:23:32Z",
    "message": "Node foo-worker-infra-3-0-kjbxr is reporting: \"exit status 1\"",
    "reason": "1 nodes are reporting degraded status on sync",
    "status": "True",
    "type": "NodeDegraded"
  },
  {
    "lastTransitionTime": "2023-07-04T08:23:32Z",
    "message": "",
    "reason": "",
    "status": "True",
    "type": "Degraded"
  }
]

No error being reported and rollout of MachineConfig respectively OpenShift Container Platform 4 update to complete without any issue being reported.