After upgrade to OpenShift Container Platform 4.13.4, MachineConfigPool degraded as MachineConfig update fails to complete

After update to OpenShift Container Platform 4.13.4 the MachineConfigPool is found in degraded state because the MachineConfigDaemon is failing to complete the upgrade and is actually stuck in a loop (post OpenShift Container Platform 4 - Node reboot).

2023-07-04T08:31:29.132976399Z I0704 08:31:29.132963    2820 update.go:1342] Removed stale file "/etc/sysctl.d/enable-userfaultfd.conf"
2023-07-04T08:31:29.133011208Z I0704 08:31:29.133002    2820 update.go:2055] Removing SIGTERM protection
2023-07-04T08:31:29.133025668Z E0704 08:31:29.133014    2820 writer.go:200] Marking Degraded due to: exit status 1
2023-07-04T08:32:29.146841055Z I0704 08:32:29.146781    2820 daemon.go:582] State and Reason: Degraded exit status 1
2023-07-04T08:32:30.761725462Z I0704 08:32:30.761681    2820 rpm-ostree.go:382] Running captured: rpm-ostree kargs
2023-07-04T08:32:30.870536786Z I0704 08:32:30.870487    2820 daemon.go:666] Preflight config drift check successful (took 1.72147513s)
2023-07-04T08:32:30.870536786Z I0704 08:32:30.870516    2820 update.go:2047] Adding SIGTERM protection
2023-07-04T08:32:30.917766296Z I0704 08:32:30.917725    2820 update.go:500] Checking Reconcilable for config rendered-foo-worker-infra-8043a0ad5ba74d790e9406980e180bc2 to rendered-foo-worker-infra-dd1d2c328b322572f3eefd2e547ba8d4
2023-07-04T08:32:31.015552082Z I0704 08:32:31.015511    2820 update.go:2025] Starting update from rendered-foo-worker-infra-8043a0ad5ba74d790e9406980e180bc2 to rendered-foo-worker-infra-dd1d2c328b322572f3eefd2e547ba8d4: &{osUpdate:true kargs:false fips:false passwd:false files:true units:true kernelType:false extensions:false}
2023-07-04T08:32:31.017042947Z I0704 08:32:31.017012    2820 helpers.go:996] File diff: /etc/sysctl.d/enable-userfaultfd.conf was added
2023-07-04T08:32:31.017042947Z I0704 08:32:31.017039    2820 helpers.go:1000] File diff: detected change to /etc/kubernetes/kubelet-ca.crt
2023-07-04T08:32:31.017062848Z I0704 08:32:31.017052    2820 helpers.go:1000] File diff: detected change to /etc/crio/crio.conf.d/00-default
2023-07-04T08:32:31.017095290Z I0704 08:32:31.017084    2820 update.go:2025] drain is already completed on this node
2023-07-04T08:32:31.018306989Z I0704 08:32:31.018293    2820 update.go:1213] Updating files
2023-07-04T08:32:31.018306989Z I0704 08:32:31.018301    2820 file_writers.go:163] Writing file "/usr/local/bin/nm-clean-initrd-state.sh"
2023-07-04T08:32:31.021545953Z I0704 08:32:31.021522    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/01-ipv6.conf"
2023-07-04T08:32:31.023929555Z I0704 08:32:31.023911    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/20-keyfiles.conf"
2023-07-04T08:32:31.026312448Z I0704 08:32:31.026290    2820 file_writers.go:163] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
2023-07-04T08:32:31.028802610Z I0704 08:32:31.028784    2820 file_writers.go:163] Writing file "/etc/audit/rules.d/mco-audit-quiet-containers.rules"
2023-07-04T08:32:31.031142225Z I0704 08:32:31.031126    2820 file_writers.go:163] Writing file "/etc/tmpfiles.d/cleanup-cni.conf"
2023-07-04T08:32:31.033510457Z I0704 08:32:31.033486    2820 file_writers.go:163] Writing file "/etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem"
2023-07-04T08:32:31.035047768Z I0704 08:32:31.035033    2820 file_writers.go:163] Writing file "/usr/local/bin/configure-ovs.sh"
2023-07-04T08:32:31.038911142Z I0704 08:32:31.038887    2820 file_writers.go:163] Writing file "/etc/containers/storage.conf"
2023-07-04T08:32:31.041281815Z I0704 08:32:31.041262    2820 file_writers.go:163] Writing file "/etc/mco/proxy.env"
2023-07-04T08:32:31.043616520Z I0704 08:32:31.043596    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/10-default-env-godebug.conf"
2023-07-04T08:32:31.046045132Z I0704 08:32:31.046026    2820 file_writers.go:163] Writing file "/etc/modules-load.d/iptables.conf"
2023-07-04T08:32:31.048663171Z I0704 08:32:31.048643    2820 file_writers.go:163] Writing file "/etc/node-sizing-enabled.env"
2023-07-04T08:32:31.051454995Z I0704 08:32:31.051433    2820 file_writers.go:163] Writing file "/usr/local/sbin/dynamic-system-reserved-calc.sh"
2023-07-04T08:32:31.054159138Z I0704 08:32:31.054139    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/kubelet-cgroups.conf"
2023-07-04T08:32:31.056602335Z I0704 08:32:31.056582    2820 file_writers.go:163] Writing file "/etc/systemd/system/kubelet.service.d/20-logging.conf"
2023-07-04T08:32:31.060440920Z I0704 08:32:31.060417    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/sdn.conf"
2023-07-04T08:32:31.063127248Z I0704 08:32:31.063102    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh"
2023-07-04T08:32:31.066955881Z I0704 08:32:31.066930    2820 file_writers.go:163] Writing file "/var/lib/kubelet/config.json"
2023-07-04T08:32:31.069906347Z I0704 08:32:31.069886    2820 file_writers.go:163] Writing file "/etc/kubernetes/ca.crt"
2023-07-04T08:32:31.072897180Z I0704 08:32:31.072880    2820 file_writers.go:163] Writing file "/etc/sysctl.d/arp.conf"
2023-07-04T08:32:31.075549695Z I0704 08:32:31.075533    2820 file_writers.go:163] Writing file "/etc/sysctl.d/forward.conf"
2023-07-04T08:32:31.078204914Z I0704 08:32:31.078186    2820 file_writers.go:163] Writing file "/etc/sysctl.d/inotify.conf"
2023-07-04T08:32:31.080799835Z I0704 08:32:31.080779    2820 file_writers.go:163] Writing file "/etc/sysctl.d/enable-userfaultfd.conf"
2023-07-04T08:32:31.085621738Z I0704 08:32:31.085591    2820 file_writers.go:163] Writing file "/etc/sysctl.d/vm-max-map.conf"
2023-07-04T08:32:31.088312397Z I0704 08:32:31.088292    2820 file_writers.go:163] Writing file "/usr/local/bin/mco-hostname"
2023-07-04T08:32:31.090875202Z I0704 08:32:31.090857    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet-plugins/volume/exec/.dummy"
2023-07-04T08:32:31.092065448Z I0704 08:32:31.092041    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
2023-07-04T08:32:31.094580986Z I0704 08:32:31.094561    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf"
2023-07-04T08:32:31.097009942Z I0704 08:32:31.096987    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/00-default"
2023-07-04T08:32:31.099577945Z I0704 08:32:31.099557    2820 file_writers.go:163] Writing file "/etc/containers/policy.json"
2023-07-04T08:32:31.101999510Z I0704 08:32:31.101982    2820 file_writers.go:163] Writing file "/etc/kubernetes/cloud.conf"
2023-07-04T08:32:31.104461465Z I0704 08:32:31.104442    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet.conf"
2023-07-04T08:32:31.107166866Z I0704 08:32:31.107149    2820 file_writers.go:163] Writing file "/usr/local/bin/kubenswrapper"
2023-07-04T08:32:31.109576732Z I0704 08:32:31.109556    2820 file_writers.go:163] Writing file "/etc/chrony.conf"
2023-07-04T08:32:31.112003946Z I0704 08:32:31.111984    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/01-mc-addInheritableCapabilities"
2023-07-04T08:32:31.114783721Z I0704 08:32:31.114759    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf.d/01-image-searchRegistries.conf"
2023-07-04T08:32:31.117886388Z I0704 08:32:31.117866    2820 file_writers.go:223] Writing systemd unit "NetworkManager-clean-initrd-state.service"
2023-07-04T08:32:31.120337881Z I0704 08:32:31.120321    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:31.123435190Z I0704 08:32:31.123415    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.123435190Z I0704 08:32:31.123430    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-profile-unix-socket.conf"
2023-07-04T08:32:31.125857113Z I0704 08:32:31.125832    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:31.390486704Z I0704 08:32:31.390445    2820 update.go:1483] Preset systemd unit crio.service
2023-07-04T08:32:31.390486704Z I0704 08:32:31.390472    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:31.398987268Z I0704 08:32:31.398945    2820 update.go:1520] Could not reset unit preset for docker.socket, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file docker.socket does not exist.
2023-07-04T08:32:31.398987268Z )
2023-07-04T08:32:31.398987268Z I0704 08:32:31.398960    2820 file_writers.go:223] Writing systemd unit "kubelet-auto-node-size.service"
2023-07-04T08:32:31.401587733Z I0704 08:32:31.401571    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:31.404448091Z I0704 08:32:31.404434    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.404448091Z I0704 08:32:31.404443    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:31.406920375Z I0704 08:32:31.406906    2820 file_writers.go:223] Writing systemd unit "kubelet.service"
2023-07-04T08:32:31.409311616Z I0704 08:32:31.409298    2820 file_writers.go:223] Writing systemd unit "kubens.service"
2023-07-04T08:32:31.411446522Z I0704 08:32:31.411433    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-firstboot.service"
2023-07-04T08:32:31.413867275Z I0704 08:32:31.413833    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-pull.service"
2023-07-04T08:32:31.416563042Z I0704 08:32:31.416531    2820 file_writers.go:223] Writing systemd unit "node-valid-hostname.service"
2023-07-04T08:32:31.419057272Z I0704 08:32:31.419028    2820 file_writers.go:223] Writing systemd unit "nodeip-configuration.service"
2023-07-04T08:32:31.421387560Z I0704 08:32:31.421363    2820 file_writers.go:223] Writing systemd unit "ovs-configuration.service"
2023-07-04T08:32:31.423804731Z I0704 08:32:31.423785    2820 file_writers.go:137] Writing systemd unit dropin "10-ovs-vswitchd-restart.conf"
2023-07-04T08:32:31.697822673Z I0704 08:32:31.697776    2820 update.go:1483] Preset systemd unit ovs-vswitchd.service
2023-07-04T08:32:31.697888155Z I0704 08:32:31.697872    2820 file_writers.go:137] Writing systemd unit dropin "10-ovsdb-restart.conf"
2023-07-04T08:32:31.701610253Z I0704 08:32:31.701579    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.706806366Z I0704 08:32:31.706775    2820 update.go:1520] Could not reset unit preset for pivot.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file pivot.service does not exist.
2023-07-04T08:32:31.706806366Z )
2023-07-04T08:32:31.706806366Z I0704 08:32:31.706793    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:31.976103084Z I0704 08:32:31.976063    2820 update.go:1483] Preset systemd unit rpm-ostreed.service
2023-07-04T08:32:31.976103084Z I0704 08:32:31.976085    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:31.984458031Z I0704 08:32:31.984421    2820 update.go:1520] Could not reset unit preset for zincati.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file zincati.service does not exist.
2023-07-04T08:32:31.984458031Z )
2023-07-04T08:32:32.248022810Z I0704 08:32:32.247979    2820 update.go:1461] Enabled systemd units: [NetworkManager-clean-initrd-state.service kubelet-auto-node-size.service kubelet.service machine-config-daemon-firstboot.service machine-config-daemon-pull.service node-valid-hostname.service openvswitch.service ovs-configuration.service ovsdb-server.service]
2023-07-04T08:32:32.521333265Z I0704 08:32:32.521285    2820 update.go:1472] Disabled systemd units [kubens.service nodeip-configuration.service]
2023-07-04T08:32:32.521333265Z I0704 08:32:32.521312    2820 update.go:1279] Deleting stale data
2023-07-04T08:32:32.521443531Z I0704 08:32:32.521430    2820 update.go:1580] Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
2023-07-04T08:32:32.521443531Z I0704 08:32:32.521440    2820 update.go:1547] Creating missing SSH key dir at /home/core/.ssh/authorized_keys.d
2023-07-04T08:32:32.529818660Z I0704 08:32:32.529792    2820 update.go:1213] Updating files
2023-07-04T08:32:32.529818660Z I0704 08:32:32.529805    2820 file_writers.go:163] Writing file "/usr/local/bin/nm-clean-initrd-state.sh"
2023-07-04T08:32:32.532664744Z I0704 08:32:32.532642    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/01-ipv6.conf"
2023-07-04T08:32:32.535250673Z I0704 08:32:32.535226    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/20-keyfiles.conf"
2023-07-04T08:32:32.537906978Z I0704 08:32:32.537885    2820 file_writers.go:163] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
2023-07-04T08:32:32.540446432Z I0704 08:32:32.540426    2820 file_writers.go:163] Writing file "/etc/audit/rules.d/mco-audit-quiet-containers.rules"
2023-07-04T08:32:32.543767171Z I0704 08:32:32.543744    2820 file_writers.go:163] Writing file "/etc/tmpfiles.d/cleanup-cni.conf"
2023-07-04T08:32:32.546217109Z I0704 08:32:32.546200    2820 file_writers.go:163] Writing file "/etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem"
2023-07-04T08:32:32.547950869Z I0704 08:32:32.547936    2820 file_writers.go:163] Writing file "/usr/local/bin/configure-ovs.sh"
2023-07-04T08:32:32.552111397Z I0704 08:32:32.552090    2820 file_writers.go:163] Writing file "/etc/containers/storage.conf"
2023-07-04T08:32:32.554524136Z I0704 08:32:32.554507    2820 file_writers.go:163] Writing file "/etc/mco/proxy.env"
2023-07-04T08:32:32.557275961Z I0704 08:32:32.557248    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/10-default-env-godebug.conf"
2023-07-04T08:32:32.559600586Z I0704 08:32:32.559582    2820 file_writers.go:163] Writing file "/etc/modules-load.d/iptables.conf"
2023-07-04T08:32:32.562101524Z I0704 08:32:32.562064    2820 file_writers.go:163] Writing file "/etc/node-sizing-enabled.env"
2023-07-04T08:32:32.564682417Z I0704 08:32:32.564661    2820 file_writers.go:163] Writing file "/usr/local/sbin/dynamic-system-reserved-calc.sh"
2023-07-04T08:32:32.567178788Z I0704 08:32:32.567159    2820 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/kubelet-cgroups.conf"
2023-07-04T08:32:32.569635827Z I0704 08:32:32.569612    2820 file_writers.go:163] Writing file "/etc/systemd/system/kubelet.service.d/20-logging.conf"
2023-07-04T08:32:32.572151315Z I0704 08:32:32.572130    2820 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/sdn.conf"
2023-07-04T08:32:32.574782788Z I0704 08:32:32.574709    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh"
2023-07-04T08:32:32.577763285Z I0704 08:32:32.577741    2820 file_writers.go:163] Writing file "/var/lib/kubelet/config.json"
2023-07-04T08:32:32.580253818Z I0704 08:32:32.580237    2820 file_writers.go:163] Writing file "/etc/kubernetes/ca.crt"
2023-07-04T08:32:32.582696360Z I0704 08:32:32.582681    2820 file_writers.go:163] Writing file "/etc/sysctl.d/arp.conf"
2023-07-04T08:32:32.585085435Z I0704 08:32:32.585068    2820 file_writers.go:163] Writing file "/etc/sysctl.d/forward.conf"
2023-07-04T08:32:32.588642085Z I0704 08:32:32.588624    2820 file_writers.go:163] Writing file "/etc/sysctl.d/inotify.conf"
2023-07-04T08:32:32.594399117Z I0704 08:32:32.594381    2820 file_writers.go:163] Writing file "/etc/sysctl.d/vm-max-map.conf"
2023-07-04T08:32:32.597034484Z I0704 08:32:32.597013    2820 file_writers.go:163] Writing file "/usr/local/bin/mco-hostname"
2023-07-04T08:32:32.599620615Z I0704 08:32:32.599602    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet-plugins/volume/exec/.dummy"
2023-07-04T08:32:32.600955688Z I0704 08:32:32.600919    2820 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
2023-07-04T08:32:32.603456084Z I0704 08:32:32.603439    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf"
2023-07-04T08:32:32.606369537Z I0704 08:32:32.606350    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/00-default"
2023-07-04T08:32:32.610556300Z I0704 08:32:32.610539    2820 file_writers.go:163] Writing file "/etc/containers/policy.json"
2023-07-04T08:32:32.613126991Z I0704 08:32:32.613110    2820 file_writers.go:163] Writing file "/etc/kubernetes/cloud.conf"
2023-07-04T08:32:32.615683508Z I0704 08:32:32.615630    2820 file_writers.go:163] Writing file "/etc/kubernetes/kubelet.conf"
2023-07-04T08:32:32.618887956Z I0704 08:32:32.618837    2820 file_writers.go:163] Writing file "/usr/local/bin/kubenswrapper"
2023-07-04T08:32:32.621550528Z I0704 08:32:32.621534    2820 file_writers.go:163] Writing file "/etc/chrony.conf"
2023-07-04T08:32:32.624263775Z I0704 08:32:32.624245    2820 file_writers.go:163] Writing file "/etc/crio/crio.conf.d/01-mc-addInheritableCapabilities"
2023-07-04T08:32:32.627698830Z I0704 08:32:32.627642    2820 file_writers.go:163] Writing file "/etc/containers/registries.conf.d/01-image-searchRegistries.conf"
2023-07-04T08:32:32.630200731Z I0704 08:32:32.630184    2820 file_writers.go:223] Writing systemd unit "NetworkManager-clean-initrd-state.service"
2023-07-04T08:32:32.632449548Z I0704 08:32:32.632433    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:32.635029500Z I0704 08:32:32.635012    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:32.635029500Z I0704 08:32:32.635022    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-profile-unix-socket.conf"
2023-07-04T08:32:32.637234367Z I0704 08:32:32.637180    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:32.904509018Z I0704 08:32:32.904452    2820 update.go:1483] Preset systemd unit crio.service
2023-07-04T08:32:32.904509018Z I0704 08:32:32.904473    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:32.913062652Z I0704 08:32:32.913026    2820 update.go:1520] Could not reset unit preset for docker.socket, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file docker.socket does not exist.
2023-07-04T08:32:32.913062652Z )
2023-07-04T08:32:32.913140214Z I0704 08:32:32.913110    2820 file_writers.go:223] Writing systemd unit "kubelet-auto-node-size.service"
2023-07-04T08:32:32.915969332Z I0704 08:32:32.915952    2820 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
2023-07-04T08:32:32.919073945Z I0704 08:32:32.919051    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:32.919073945Z I0704 08:32:32.919061    2820 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
2023-07-04T08:32:32.922111393Z I0704 08:32:32.922093    2820 file_writers.go:223] Writing systemd unit "kubelet.service"
2023-07-04T08:32:32.925123428Z I0704 08:32:32.925099    2820 file_writers.go:223] Writing systemd unit "kubens.service"
2023-07-04T08:32:32.927479322Z I0704 08:32:32.927461    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-firstboot.service"
2023-07-04T08:32:32.931479040Z I0704 08:32:32.931460    2820 file_writers.go:223] Writing systemd unit "machine-config-daemon-pull.service"
2023-07-04T08:32:32.933850616Z I0704 08:32:32.933834    2820 file_writers.go:223] Writing systemd unit "node-valid-hostname.service"
2023-07-04T08:32:32.936694241Z I0704 08:32:32.936674    2820 file_writers.go:223] Writing systemd unit "nodeip-configuration.service"
2023-07-04T08:32:32.939181286Z I0704 08:32:32.939165    2820 file_writers.go:223] Writing systemd unit "ovs-configuration.service"
2023-07-04T08:32:32.941906819Z I0704 08:32:32.941886    2820 file_writers.go:137] Writing systemd unit dropin "10-ovs-vswitchd-restart.conf"
2023-07-04T08:32:33.208984072Z I0704 08:32:33.208942    2820 update.go:1483] Preset systemd unit ovs-vswitchd.service
2023-07-04T08:32:33.208984072Z I0704 08:32:33.208961    2820 file_writers.go:137] Writing systemd unit dropin "10-ovsdb-restart.conf"
2023-07-04T08:32:33.212118544Z I0704 08:32:33.212091    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:33.217372730Z I0704 08:32:33.217342    2820 update.go:1520] Could not reset unit preset for pivot.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file pivot.service does not exist.
2023-07-04T08:32:33.217372730Z )
2023-07-04T08:32:33.217372730Z I0704 08:32:33.217355    2820 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
2023-07-04T08:32:33.478951518Z I0704 08:32:33.478908    2820 update.go:1483] Preset systemd unit rpm-ostreed.service
2023-07-04T08:32:33.478951518Z I0704 08:32:33.478931    2820 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
2023-07-04T08:32:33.487605830Z I0704 08:32:33.487563    2820 update.go:1520] Could not reset unit preset for zincati.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file zincati.service does not exist.
2023-07-04T08:32:33.487605830Z )
2023-07-04T08:32:33.753396023Z I0704 08:32:33.753350    2820 update.go:1461] Enabled systemd units: [NetworkManager-clean-initrd-state.service kubelet-auto-node-size.service kubelet.service machine-config-daemon-firstboot.service machine-config-daemon-pull.service node-valid-hostname.service openvswitch.service ovs-configuration.service ovsdb-server.service]
2023-07-04T08:32:34.016390168Z I0704 08:32:34.016338    2820 update.go:1472] Disabled systemd units [kubens.service nodeip-configuration.service]
2023-07-04T08:32:34.016390168Z I0704 08:32:34.016367    2820 update.go:1279] Deleting stale data
2023-07-04T08:32:34.016470522Z I0704 08:32:34.016456    2820 update.go:1342] Removed stale file "/etc/sysctl.d/enable-userfaultfd.conf"
2023-07-04T08:32:34.016514286Z I0704 08:32:34.016496    2820 update.go:2055] Removing SIGTERM protection
2023-07-04T08:32:34.016523204Z E0704 08:32:34.016517    2820 writer.go:200] Marking Degraded due to: exit status 1
2023-07-04T08:33:34.036229992Z I0704 08:33:34.036176    2820 daemon.go:582] State and Reason: Degraded exit status 1

Above is the loop detailed, found in the respective machine-config-daemon pod. It basically works as the upgrade is properly triggered and running but once the system is come up with the latest version, the machine-config-daemon is stuck in the given loop.

When deleting the problematic Machine, a new Machine is being created and successfully able to join the OpenShift Container Platform 4 - Cluster without similar issues reported in machine-config-daemon.

OpenShift Container Platform 4.13.4

N/A

1. N/A - only see on one specific OpenShift Container Platform 4.13.4 - Cluster

$ omc get mcp foo-worker-infra -o json | jq '.status.conditions'
[
  {
    "lastTransitionTime": "2023-02-20T14:06:09Z",
    "message": "",
    "reason": "",
    "status": "False",
    "type": "RenderDegraded"
  },
  {
    "lastTransitionTime": "2023-06-23T05:08:08Z",
    "message": "",
    "reason": "",
    "status": "False",
    "type": "Updated"
  },
  {
    "lastTransitionTime": "2023-06-23T05:08:08Z",
    "message": "All nodes are updating to rendered-foo-worker-infra-dd1d2c328b322572f3eefd2e547ba8d4",
    "reason": "",
    "status": "True",
    "type": "Updating"
  },
  {
    "lastTransitionTime": "2023-07-04T08:23:32Z",
    "message": "Node foo-worker-infra-3-0-kjbxr is reporting: \"exit status 1\"",
    "reason": "1 nodes are reporting degraded status on sync",
    "status": "True",
    "type": "NodeDegraded"
  },
  {
    "lastTransitionTime": "2023-07-04T08:23:32Z",
    "message": "",
    "reason": "",
    "status": "True",
    "type": "Degraded"
  }
]

No error being reported and rollout of MachineConfig respectively OpenShift Container Platform 4 update to complete without any issue being reported.