After upgrading the OS and rebooting the node, some pods may not come up and the ones that fail are different every time. The replica controller doesn't kill the failed pod and start a new one. It's necessary to kill the pod manually to bring it up again. Customer suggested to raise it as critical as this is happening after every reboot.

microshift - RC3

Always

1. Rebooting
2. Run journalctl

Pods failing and not restarting

Pods starting automatically

Jan 17 14:39:11 edgenius microshift[2779]: kubelet E0117 14:39:11.799237    2779 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/topolvm.io^29dee98b-c64a-473b-b125-b83a6bc97f73 podName: nodeName:}" failed. No retries permitted until 2023-01-17 14:39:27.799225162 +0000 UTC m=+69.004817609 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-7b5f487d-ae5a-4820-b0fe-d3149828d2a1" (UniqueName: "kubernetes.io/csi/topolvm.io^29dee98b-c64a-473b-b125-b83a6bc97f73") pod "edgeinfomodeldb-5fbd9887d8-hdm8j" (UID: "20676429-968e-49f3-81fe-4cf06a875c4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name topolvm.io not found in the list of registered CSI drivers

Jan 17 14:39:11 edgenius microshift[2779]: kubelet E0117 14:39:11.800872    2779 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/topolvm.io^2a35518d-487d-498b-91d8-e03848d9d11d podName: nodeName:}" failed. No retries permitted until 2023-01-17 14:39:27.800848382 +0000 UTC m=+69.006440829 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-7a2b2701-6804-4389-89bd-96ed13312f78" (UniqueName: "kubernetes.io/csi/topolvm.io^2a35518d-487d-498b-91d8-e03848d9d11d") pod "edgeauthenticationserver-77667796cc-wz8px" (UID: "99254a2a-f130-4eaa-bae0-8f40af8082d8") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name topolvm.io not found in the list of registered CSI drivers

Jan 17 14:39:11 edgenius microshift[2779]: kubelet E0117 14:39:11.802407    2779 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/topolvm.io^44ce8ef5-2ca7-4cb7-aadb-45cb3fce179b podName: nodeName:}" failed. No retries permitted until 2023-01-17 14:39:27.802375101 +0000 UTC m=+69.007967648 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-a8fb267b-946e-4e11-a0ae-7e7801c4a529" (UniqueName: "kubernetes.io/csi/topolvm.io^44ce8ef5-2ca7-4cb7-aadb-45cb3fce179b") pod "edgetyperegistrydb-56b49789c7-xrvhf" (UID: "daea4ebe-2c14-4dc4-83de-a4c37d005b23") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name topolvm.io not found in the list of registered CSI drivers