Description of problem:
We're testing upgrades from 4.9.41 -> 4.10.21 -> 4.11.0-rc.0 -> 4.11.0-rc.1

Failure is seen during upgrade from 4.11.0-rc.0 to 4.11.0-rc.1. The 4.9 -> 4.10 -> 4.11-rc.0 upgrades had no issues.

Must gather logs: https://drive.google.com/file/d/1vLMStKyA6z1yyRYwOV9eUaPU53rVo8Pw/view?usp=sharing

The error we're seeing in mcp is:
message: 'Node master-0 is reporting: "unexpected on-disk state validating against
rendered-master-c75d108f5b1bd9301a67a1fbd8be19cc: expected target osImageURL
\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e657044831a9bd296b20d37b693698adb1d7eb2d3cd9090db7724779ecbf608\",
have \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74b8a768d5b354263cab962106cc7faa391014acfd4897356de625667ebac5ba\""'
reason: 1 nodes are reporting degraded status on sync

message: 'Node worker-1 is reporting: "unexpected on-disk state validating against
rendered-worker-44da19ca71d1d08ed7aa28e94b7421e1: expected target osImageURL
\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e657044831a9bd296b20d37b693698adb1d7eb2d3cd9090db7724779ecbf608\",
have \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74b8a768d5b354263cab962106cc7faa391014acfd4897356de625667ebac5ba\""'
reason: 1 nodes are reporting degraded status on sync
status: "True"
type: NodeDegraded

and the rpm-ostree status command shows No space left on device. The node has 120GB which is the same size always used and no issues have been seen previously.

sh-4.4# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
error: Installing kernel: regfile copy: No space left on device
check `journalctl -b -1 -u ostree-finalize-staged.service`
Deployments:

• pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74b8a768d5b354263cab962106cc7faa391014acfd4897356de625667ebac5ba
  CustomOrigin: Managed by machine-config-operator
  Version: 411.86.202206280018-0 (2022-06-28T00:25:24Z)

pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:737dda6647e7925bbe4fb5e4b8cddebedcf83c8bc233fee38bb4c5cdd97a1cd5
CustomOrigin: Managed by machine-config-operator
Version: 410.84.202206240419-0 (2022-06-24T04:25:30Z)

Version-Release number of selected component (if applicable):

1. oc version
   Client Version: 4.9.41
   Server Version: 4.11.0-rc.1
   Kubernetes Version: v1.24.0+2dd8bb1

Steps to Reproduce:
1.Perform upgrades from 4.9.41 -> 4.10.21 -> 4.11.0-rc.0 -> 4.11.0-rc.1
4.9.41 -> 4.10.21 -> 4.11.0-rc.0 was successful but failure seen when upgrading to 4.11.0-rc.1

Actual results:
ClusterVersion: Updating to "4.11.0-rc.1" from "4.11.0-rc.0" for 11 hours: Unable to apply 4.11.0-rc.1: an unknown error has occurred: MultipleErrors

Expected results:
Upgrade should be successful

Additional info:

1. oc get clusterversion
   NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
   version 4.11.0-rc.0 True True 10h Unable to apply 4.11.0-rc.1: an unknown error has occurred: MultipleErrors

1. oc get nodes
   NAME STATUS ROLES AGE VERSION
   master-0 Ready,SchedulingDisabled master 2d8h v1.24.0+9ddc8b1
   master-1 Ready master 2d8h v1.24.0+9ddc8b1
   master-2 Ready master 2d8h v1.24.0+9ddc8b1
   worker-0 Ready worker 2d8h v1.24.0+9ddc8b1
   worker-1 Ready,SchedulingDisabled worker 2d8h v1.24.0+9ddc8b1

1. oc get co
   NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
   authentication 4.11.0-rc.1 True False True 130m APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()...
   baremetal 4.11.0-rc.1 True False False 2d8h
   cloud-controller-manager 4.11.0-rc.1 True False False 2d8h
   cloud-credential 4.11.0-rc.1 True False False 2d8h
   cluster-autoscaler 4.11.0-rc.1 True False False 2d8h
   config-operator 4.11.0-rc.1 True False False 2d8h
   console 4.11.0-rc.1 True False False 9h
   csi-snapshot-controller 4.11.0-rc.1 True False False 2d2h
   dns 4.11.0-rc.1 True False False 2d8h
   etcd 4.11.0-rc.1 True False False 2d8h
   image-registry 4.11.0-rc.1 True False False 10h
   ingress 4.11.0-rc.1 True False True 29h The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-56d77b4f4d-b9mvv" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 1 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }

   , 2 node(s) were unschedulable, 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint

   {node-role.kubernetes.io/master: }

   . preemption: 0/5 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.), DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 1/2 of replicas are available)
   insights 4.11.0-rc.1 True False False 2d8h
   kube-apiserver 4.11.0-rc.1 True False False 2d8h
   kube-controller-manager 4.11.0-rc.1 True False False 2d8h
   kube-scheduler 4.11.0-rc.1 True False False 2d8h
   kube-storage-version-migrator 4.11.0-rc.1 True False False 10h
   machine-api 4.11.0-rc.1 True False False 2d8h
   machine-approver 4.11.0-rc.1 True False False 2d8h
   machine-config 4.11.0-rc.0 True True True 9h Unable to apply 4.11.0-rc.1: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool worker is not ready, retrying. Status: (pool degraded: true total: 2, ready 0, updated: 0, unavailable: 1)]
   marketplace 4.11.0-rc.1 True False False 2d8h
   monitoring 4.11.0-rc.1 False True True 9h Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
   network 4.11.0-rc.1 True False False 2d8h
   node-tuning 4.11.0-rc.1 True False False 10h
   openshift-apiserver 4.11.0-rc.1 True False True 2d8h APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()
   openshift-controller-manager 4.11.0-rc.1 True False False 2d8h
   openshift-samples 4.11.0-rc.1 True False False 10h
   operator-lifecycle-manager 4.11.0-rc.1 True False False 2d8h
   operator-lifecycle-manager-catalog 4.11.0-rc.1 True False False 2d8h
   operator-lifecycle-manager-packageserver 4.11.0-rc.1 True False False 30h
   service-ca 4.11.0-rc.1 True False False 2d8h
   storage 4.11.0-rc.1 True False False

1. oc get mcp
   NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
   master rendered-master-679e46b83af12a9f90a61c64f086cbc6 False True True 3 0 0 1 2d8h
   worker rendered-worker-ce0f663b8f5c37a8bde873ecbb8311e8 False True True 2 0 0 1 2d8h

1. oc get mcp master -oyaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
   creationTimestamp: "2022-07-04T08:45:58Z"
   generation: 7
   labels:
   machineconfiguration.openshift.io/mco-built-in: ""
   operator.machineconfiguration.openshift.io/required-for-upgrade: ""
   pools.operator.machineconfiguration.openshift.io/master: ""
   name: master
   resourceVersion: "1360931"
   uid: 7ffe8daa-08de-4abb-b38d-55db9a5adee8
   spec:
   configuration:
   name: rendered-master-c75d108f5b1bd9301a67a1fbd8be19cc
   source:

• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 00-master
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 01-master-container-runtime
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 01-master-kubelet
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-chrony-configuration
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-generated-crio-seccomp-use-default
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-generated-registries
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-ssh
  machineConfigSelector:
  matchLabels:
  machineconfiguration.openshift.io/role: master
  nodeSelector:
  matchLabels:
  node-role.kubernetes.io/master: ""
  paused: false
  status:
  conditions:
• lastTransitionTime: "2022-07-04T08:47:31Z"
  message: ""
  reason: ""
  status: "False"
  type: RenderDegraded
• lastTransitionTime: "2022-07-06T07:09:12Z"
  message: ""
  reason: ""
  status: "False"
  type: Updated
• lastTransitionTime: "2022-07-06T07:09:12Z"
  message: All nodes are updating to rendered-master-c75d108f5b1bd9301a67a1fbd8be19cc
  reason: ""
  status: "True"
  type: Updating
• lastTransitionTime: "2022-07-06T07:16:11Z"
  message: 'Node master-0 is reporting: "unexpected on-disk state validating against
  rendered-master-c75d108f5b1bd9301a67a1fbd8be19cc: expected target osImageURL
  \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e657044831a9bd296b20d37b693698adb1d7eb2d3cd9090db7724779ecbf608\",
  have \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74b8a768d5b354263cab962106cc7faa391014acfd4897356de625667ebac5ba\""'
  reason: 1 nodes are reporting degraded status on sync
  status: "True"
  type: NodeDegraded
• lastTransitionTime: "2022-07-06T07:16:11Z"
  message: ""
  reason: ""
  status: "True"
  type: Degraded
  configuration:
  name: rendered-master-679e46b83af12a9f90a61c64f086cbc6
  source:
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 00-master
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 01-master-container-runtime
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 01-master-kubelet
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-chrony-configuration
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-generated-crio-seccomp-use-default
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-generated-registries
• apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  name: 99-master-ssh
  degradedMachineCount: 1
  machineCount: 3
  observedGeneration: 7
  readyMachineCount: 0
  unavailableMachineCount: 1
  updatedMachineCount: 0

The master node which shows SchedulingDisabled, rpm-ostree shows "No space left on device"

sh-4.4# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
error: Installing kernel: regfile copy: No space left on device
check `journalctl -b -1 -u ostree-finalize-staged.service`
Deployments:

• pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74b8a768d5b354263cab962106cc7faa391014acfd4897356de625667ebac5ba
  CustomOrigin: Managed by machine-config-operator
  Version: 411.86.202206280018-0 (2022-06-28T00:25:24Z)

pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:737dda6647e7925bbe4fb5e4b8cddebedcf83c8bc233fee38bb4c5cdd97a1cd5
CustomOrigin: Managed by machine-config-operator
Version: 410.84.202206240419-0 (2022-06-24T04:25:30Z)

1. df -h
   Filesystem Size Used Avail Use% Mounted on
   /dev/sda4 120G 29G 92G 24% /
   tmpfs 8.0G 0 8.0G 0% /sys/fs/cgroup
   devtmpfs 7.9G 0 7.9G 0% /dev
   tmpfs 8.0G 128K 8.0G 1% /dev/shm
   tmpfs 8.0G 93M 7.9G 2% /run
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/7649fea8a9f22c2fa4d7d454380c261bca15a34b3b2c75e8503919cdb65755ae/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/7dbcf0b1e7c0725f91eed644fbfa3f30c4d41165ccb43d560faf995da79b1451/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/9b39f3ef0687628c5f7d6639a18b38147f31013e455ab22b68043418c2e953c3/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/c7b6f9393e55d8270550ab8a4ccb2e42a4784623e1d2a0901f30d662a8291a5f/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/23dcaaa0cf8459be2034ef76a658782b51da69d2979c6e06f96e534dc7fce55d/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/d106800bf4ea805fe5df5d2dc6a1416f17b983d1eac08d6ca41addba72a5f2b6/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/6fd657ea28e3335de406979dd4e052f3ceef5bf62e50073fb51331dc0cd9f858/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/c0127e0d16e86b97f802921c336b2549246f967b03ce642b62e00e14dfbdcddb/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/13248f323be7115007c5797fc79374d8383fb2c824bee4e10879ed89979ae325/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/ae24555c8ac30a5689a6b2a12e2a51a69b40bddb7b69267747c194cffcc42104/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/072999891e0af7625c4a69456ad28003d94214a8929c6d631cd09062595e98e4/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/6c502811e9dc0ed12c3768964a5d5bd5a06d673265dccfc972ecc22dea816047/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/9c875c2647fb15f438654895d77b297e40a4baa4f53278499c8a12de9e5d0071/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/2067db47b62476ce457bfcb0f2b36b944d9191991fed5efa225bf2581c5e92b1/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/3f33deb9494424a0d0653dbd171243785a65a535a36960ee92924c27813a670b/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/6d68b8669f7c956a1c399e315b931087c0a306d6875d4a3f54eeece93dee90b8/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/d613e523b5ae6ce901224f4c018d8691e1a0b196f097d078d96dc5c611f64dac/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/a9db656e85403abf52c9c8b940621d92ae1c0291355cea50beb9762dc56f212e/userdata/shm
   shm 64M 0 64M 0% /run/containers/storage/overlay-containers/7849996a3ba2e0c582f1d561638bb6e79d4d7b04a109ef7893d4653a51d5ab9d/userdata/shm
   shm 64M 0 64M 0% /var/lib/containers/storage/overlay/58c70b88804fe53bb3f6abd09068d797d60b1de50718c8c9a7317336080f4cd9/merged/dev/shm
   tmpfs 8.0G 64K 8.0G 1% /tmp
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/74cbd66f09a019daa6b380fad95e1cf137588f9427fa7a09e609c17cb4913d5a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/a2bf51318544322cdab4dbdec432c6a44fdbfe1941521ded6e42de8f20dcc9a5/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/0d6a12c4cedd397ba548756a3394ea3fa0a8d92e293c74ac894a69bd255bfd8a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/207d6350ff9cddc1bc927d7246ccf1cdf8d4d2a5da8d4e7fda40fc7f5c586eb1/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/35fa96660585c22869604bc2adfe7ba5dae5e98264248a22618ba3a77b9b01c8/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/c78ba99d0466bbb0fdc527097516bc3afa394b976d1af2bf98f6a1a00f0d36d5/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/44e54c4bdc8256882dc6c552ff9bcd13b14c8c9137d5735685c877bae0b16898/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/9246569565c2f6853fbae1459e0bdfde24dc70ee705f2b7959ca3a4d580f8614/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/d1784f9508fd316e038f3de86a3a0d94671710978b442f7320a6dba35c59d097/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/7d49aa9d690330efbca62b29382d8b98ac37c212ba246991f919a8f7122a8f9a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/810a309f9c28435dccf28b6bbf2e0b478987da168f302d5c38ffc63cec0cca1d/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/4ebd116b4fd0c1b6dcfee481ee99e79e6f58363a3339ba8a6e9856afc91ada1d/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/87c0507e33fdf3aee4bf360e1dfd6e8cac8d47c4366c415c8cb5223e15818cda/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/36e44bf7bd05c1f29d09f6b5631813113d5d7c7ba9b1974fd1babf18682e0ce7/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/6d64637eaa711cb4adccd09d798176088e2b5c5c350216bdb81913ef28c5a26a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/1e42f32a853da1d1aa69064d87e1ba80a30512d101b60c8032e954809a50e2e3/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/38bcadb922b144e48d3c2468d4d907b1003271d7827ecde6d0152642a7aff9dd/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/8860a2acce06c63616753f70297dc6f7a535dc4b057619b2f9c3d54306e28f11/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/219f50bd78b55073f67c07d5c42f61504b694baae4c4b2a71729a561ae7dc754/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/76a57b9167e99079d28d8cbd1d4b6c272dbc739dec5d14aa35d4f9595eb56fe7/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/db5c1db06cacb60a21d7a837b1b2dd95ce649b01c0fa815a74784d7a2770ab4b/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/617ca0bf2b66921ab96c449b36cd5795993f699148b39bfc21c8c8b0d7182463/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/d7825f691375e46d90e8a4f8f274d991714851f1b28de8adda3509d6cae5582e/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/7fcad1d9aedd38ce29648ed42ceb2700af56269f00f48a1e1075fb5a6ae791d4/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/a5bba405f1ade715b1c1c262d8a9fc1996260bb2a57f72d591105195ff053ff0/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/b597c10fa290a32262f1a234f0c3459006db360dd59fffcb8ad018e1e13481d1/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/5cd0896a0cea72fe0d9d8f5e4541ac85d04c0bc7223458c6e0ecd28b429af92b/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/e113aef1a3925c4addd4f4dee2c9c4c869d13d1f4a9254abdbe9f75d73200761/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/cccf8df999427e00b0cfe9251eb42c480c82934e5c9a067fca7fc598a5fcc65a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/6a09208e93c9640981d81df25adf39e3e3aa1442c40829bb436a0d449b8e6789/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/c28a84fa2dd2d7e1b316088575a9ba95091106b3ae2fec81f1011b42b4988d76/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/d49a521d159caae9e35c5cd61027c37cedbe31d90773c353444023964fccbddc/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/2628c2ddb7cada7f45414ceec1ac7f6d838533988bd84d97bc06b280677ea90a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/2381711c2666ebce5e0d843d048b4e72451405b9938666f21f51eb782a62dd9b/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/1651cd13c13c71211bd8e1bd00b03e524184f7cd70a342d11842546c7b41a316/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/008b9c6a387186aefacb6e66efb0b62ad2522ee3e82fff13f1154743589b17ec/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/6fcf532d999fe7a8c5f650fb1732f49c1fd05d32860dbb863dc238b5efdb71df/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/0a60f82a8e4302128e9649673f5a2960bf096e4e9c1ba75684d0decefa17474e/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/1d39309225432518d77f9d88350c386a9a588f883aaefd53d82a6f47af23f3cc/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/862d52a13a97bda79f6b0034466d6fd57860dfea9491813c86faef825d5ecd6e/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/a61929c80480b2b73cd9ec4473ebf2a7bbc1679f0210b38e1c73e8156460cd9a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/5930d82feba22075c77a790431fef5e6b328d505b5dcf4722203e61cb9c08c7e/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/699a26ddb9cc963785865db12ec43df8300d9fa163de5eee2e0acfa7ef01c757/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/e4ccccb65ad4582606f0f2f3c47a0e171cad51f635aef7be1ff7d9ec1607efde/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/0f7459e495f8a413bdeca09c4f0b308c027e99233b65ff2ca1c4cc00173d4d42/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/2e98e461a51e752ea04fa6a4a05fb90f83a3a3b1b0d098138d5716890d4b8c9e/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/6e802c011f43f0365f05bdccfa3ee63f1be9aa18bcb6cf8e72e820d7be26e53c/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/570e86287b39f0243fc3e2f4f5d5de2ae9928f980690c00b7063d738bc5feb7b/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/ed6379bc5468fc0e5cd99c5d2a66d9db90d54a5d0d30627aca51845be950a6b8/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/390c28bc760426445be7e1134fac7752cff70ac131b8a8174931051b6f3ee28c/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/e39468566dc0728b058ff80692671c936636dc1d928bf1d2c2576365dcf8602e/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/94d7dbf34119f65b24d8f79cb09218ba5a14ae03ddcd4609ea00c2f1f8ec34b3/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/99671df43387654349867ad680bbd4e089430d2830fa568bb3d91d2c24deb645/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/a19026519e1fe43740b0a3aa08bd927f9d85293ff33461a61d2e83573feb5e4c/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/40e29e08a6714713ea91ab9bac89f16700d9a97dc4285b6d06e6e20e55bc9033/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/748afa3ae2a928d646c3b8168f9e210f4db662b1a344f2998a49b6b0367d7c3f/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/f12a7cbe53d3e3c439a4137e2307032c77108d527fd7affd4de38c109ac07b12/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/58af3dbb8abd5aaaa9cce160da06a1193deb8cf7b0c01992255b1a301d348675/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/bdfbd69d4745a66d8990598d97d90b8beee3bd22f90290514b3be036df5bbcf9/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/b42ae7e7e33f5ddc9700806f4aa45870984ee60d3efe834f4744017b8b934234/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/c3d0d89cea7e5429e9088a8cf4789b516841837771a1c08b7400204cd1ea0829/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/a37d1b59abe259b5cfe29dc00d3af0bd4d256eb01fa917573c1210272551a299/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/9e0601986463bdf98ed4c811c5021324330d18380e4f2b68f58e8cd2190e6adf/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/db5bbea59e6a897a0d7afb9fdb1dee3d8a9444c82b731d43f5a9511528947921/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/b75af4b99cd974864d0794f2c94f0068d0229836df659c38f36630187ad463a1/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/2810ba50f210316dbf2ec97bbc4a1b1ccfd77c2b17c7a0f8b8637ffed365e582/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/63bcdf65d9e066ea62a4068c739c715f4556a344cbb8e327670ac6470c867d2c/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/9a056c95c186e8a86c5a1d54cf38d6ee82bf150cf71711398d9950ae63e965b2/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/10216edeb11cd9086ce89ccff3b94e5c651b8ff7928b2bed06e2735e75e0d790/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/af8804ab535ee457b8b2d97e085b3e6b65278c9fa896045ffed398047f4b647a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/7b581a408892eb681cbc05c8d480cdc78a8bffe628f30efa2ce8b66909b520e3/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/d1792accc713b7f3c5e47aa77b13b4b385947a4349765cbd7f5afccd39b7d707/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/33e2a26543733f65c7eb5b3e7f8c43dac6471a8976ea9a36d4de673ddf0b74ba/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/555aab703ec997f75d71738cf30587b42fb99890d001d217fcefd3a8ff27fa80/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/4543abaef27655d54e25b08d867075f3bfa164137d130f94e9593aa32b3a7833/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/68eb567e7678693b66ff1d378bfc065fb8f50958b0158df9e06fa3526aed6d4a/merged
   overlay 120G 29G 92G 24% /var/lib/containers/storage/overlay/58c70b88804fe53bb3f6abd09068d797d60b1de50718c8c9a7317336080f4cd9/merged
   tmpfs 64M 0 64M 0% /var/lib/containers/storage/overlay/58c70b88804fe53bb3f6abd09068d797d60b1de50718c8c9a7317336080f4cd9/merged/dev
   tmpfs 8.0G 0 8.0G 0% /var/lib/containers/storage/overlay/58c70b88804fe53bb3f6abd09068d797d60b1de50718c8c9a7317336080f4cd9/merged/sys/fs/cgroup
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/45651ff9-3f73-4925-b696-8255c59bf8c2/volumes/kubernetes.io~secret/node-bootstrap-token
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/45651ff9-3f73-4925-b696-8255c59bf8c2/volumes/kubernetes.io~secret/certs
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/af9ccda7-a5f1-47fe-b359-3bab5d9a7b3f/volumes/kubernetes.io~projected/kube-api-access-92wl5
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/45651ff9-3f73-4925-b696-8255c59bf8c2/volumes/kubernetes.io~projected/kube-api-access-t5tk9
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/6145dbf0-605f-4380-9358-5f7bf971f25e/volumes/kubernetes.io~projected/kube-api-access-vclcj
   tmpfs 15G 64K 15G 1% /var/lib/kubelet/pods/6145dbf0-605f-4380-9358-5f7bf971f25e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/6145dbf0-605f-4380-9358-5f7bf971f25e/volumes/kubernetes.io~secret/node-exporter-tls
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/0d0e1319-1f01-4423-9541-90d95791a8d0/volumes/kubernetes.io~secret/sdn-controller-metrics-certs
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/5a5b814d-7d1e-49e4-8007-5b655371fc1c/volumes/kubernetes.io~secret/metrics-tls
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/58cf6b33-d5f1-496e-bd8b-fbf49c4aac73/volumes/kubernetes.io~secret/serving-cert
   tmpfs 15G 64K 15G 1% /var/lib/kubelet/pods/c7581191-e489-488b-9e17-8e148a024820/volumes/kubernetes.io~secret/cookie-secret
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/4d231826-0fe8-4bab-91a4-9380094b92e0/volumes/kubernetes.io~secret/webhook-certs
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/02c26524-bbc7-44c2-bfb0-6187b4f2abe0/volumes/kubernetes.io~secret/metrics-certs
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/c7581191-e489-488b-9e17-8e148a024820/volumes/kubernetes.io~secret/proxy-tls
   tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/91517de1-83fd-4bc1-9329-91b436f18088/volumes/kubernetes.io~secret/sdn-metrics-certs
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/02c26524-bbc7-44c2-bfb0-6187b4f2abe0/volumes/kubernetes.io~projected/kube-api-access-v82nt
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/4d231826-0fe8-4bab-91a4-9380094b92e0/volumes/kubernetes.io~projected/kube-api-access-skhct
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/c7581191-e489-488b-9e17-8e148a024820/volumes/kubernetes.io~projected/kube-api-access-kg4xg
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/fbe012b0-d7d6-4647-a29d-17e832cdbcec/volumes/kubernetes.io~projected/kube-api-access-j5hbr
   tmpfs 1.0G 256K 1.0G 1% /var/lib/kubelet/pods/996dbdf7-9f61-4d1d-82ed-44514836a980/volumes/kubernetes.io~projected/kube-api-access-5rlll
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/aa36428d-9ec2-4601-a993-f7e34a335d7e/volumes/kubernetes.io~projected/kube-api-access-qjxxj
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/91517de1-83fd-4bc1-9329-91b436f18088/volumes/kubernetes.io~projected/kube-api-access-q54ch
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/a876c756-76ac-460b-915e-43b23f5fe01e/volumes/kubernetes.io~projected/kube-api-access-qw5qz
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/df98d7d5-1ee5-4a73-8f46-cb7851b59c6a/volumes/kubernetes.io~projected/kube-api-access-xmqbr
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/0d0e1319-1f01-4423-9541-90d95791a8d0/volumes/kubernetes.io~projected/kube-api-access-kbs75
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/58cf6b33-d5f1-496e-bd8b-fbf49c4aac73/volumes/kubernetes.io~projected/kube-api-access-4wdhz
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/e49617fc-8b23-40d3-9699-6a32e460d993/volumes/kubernetes.io~projected/kube-api-access-jn2bz
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/5a5b814d-7d1e-49e4-8007-5b655371fc1c/volumes/kubernetes.io~projected/kube-api-access-kpnkr
   tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/2a00c1c3-2818-4fd7-8040-7810d495bdf6/volumes/kubernetes.io~projected/kube-api-access-lkrmz
   /dev/sda3 364M 277M 65M 82% /boot

sh-4.4# ls -lart /boot/*
lrwxrwxrwx. 1 root root 1 Oct 8 2021 /boot/boot -> .
lrwxrwxrwx. 1 root root 8 Jul 5 11:40 /boot/loader -> loader.0

/boot/lost+found:
total 14
drwx------. 2 root root 12288 Oct 8 2021 .
drwxr-xr-x. 7 root root 1024 Jul 6 07:13 ..

/boot/grub2:
total 185
drwxr-xr-x. 2 root root 8192 Oct 8 2021 powerpc-ieee1275
drwxr-xr-x. 2 root root 1024 Oct 8 2021 locale
rw-rr-. 1 root root 1024 Oct 8 2021 grubenv
-rwxr-xr-x. 1 root root 2625 Oct 8 2021 grub.cfg
rw-rr-. 1 root root 164668 Oct 8 2021 grub
drwxr-xr-x. 2 root root 1024 Oct 8 2021 fonts
drwxr-xr-x. 5 root root 1024 Oct 8 2021 .
drwxr-xr-x. 7 root root 1024 Jul 6 07:13 ..

/boot/loader.0:
total 6
drwxr-xr-x. 3 root root 1024 Jul 5 11:40 .
drwxr-xr-x. 2 root root 1024 Jul 5 11:40 entries
drwxr-xr-x. 7 root root 1024 Jul 6 07:13 ..

/boot/ostree:
total 10
drwxr-xr-x. 2 root root 1024 Jul 4 14:44 rhcos-0bfcbbf7c37c79bff574f3829112a64ecee6c084597b1d93e90bc00f07954a73
drwxr-xr-x. 2 root root 1024 Jul 5 11:40 rhcos-952c9677e944c268239aec2429e421d2a4fd4c751550a1c5f74d83bd2c5ddb1f
drwxr-xr-x. 2 root root 1024 Jul 6 07:13 rhcos-1d101bb28fbd71daf2b6ffb8cf1261576cfd670cec4604a91b20a713523370a2
drwxr-xr-x. 7 root root 1024 Jul 6 07:13 ..
drwxr-xr-x. 5 root root 1024 Jul 6 07:13 .

/boot/loader.1:
total 6
drwxr-xr-x. 2 root root 1024 Jul 6 07:13 entries
drwxr-xr-x. 7 root root 1024 Jul 6 07:13 ..
drwxr-xr-x. 3 root root 1024 Jul 6 07:13 .

Must gather logs: https://drive.google.com/file/d/1vLMStKyA6z1yyRYwOV9eUaPU53rVo8Pw/view?usp=sharing