Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-10256

Some pods not coming up after rebooting

    XMLWordPrintable

Details

    • uShift Sprint 231, uShift Sprint 232, uShift Sprint 233
    • 3
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      After upgrading the OS and rebooting the node, some pods may not come up and the ones that fail are different every time. The replica controller doesn't kill the failed pod and start a new one. It's necessary to kill the pod manually to bring it up again. Customer suggested to raise it as critical as this is happening after every reboot.

      Version-Release number of selected component (if applicable):

      microshift - RC3

      How reproducible:

      Always

      Steps to Reproduce:

      1. Rebooting
2. Run journalctl

      Actual results:

      Pods failing and not restarting

      Expected results:

      Pods starting automatically

      Additional info:

      Jan 17 14:39:11 edgenius microshift[2779]: kubelet E0117 14:39:11.799237    2779 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/topolvm.io^29dee98b-c64a-473b-b125-b83a6bc97f73 podName: nodeName:}" failed. No retries permitted until 2023-01-17 14:39:27.799225162 +0000 UTC m=+69.004817609 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-7b5f487d-ae5a-4820-b0fe-d3149828d2a1" (UniqueName: "kubernetes.io/csi/topolvm.io^29dee98b-c64a-473b-b125-b83a6bc97f73") pod "edgeinfomodeldb-5fbd9887d8-hdm8j" (UID: "20676429-968e-49f3-81fe-4cf06a875c4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name topolvm.io not found in the list of registered CSI drivers

Jan 17 14:39:11 edgenius microshift[2779]: kubelet E0117 14:39:11.800872    2779 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/topolvm.io^2a35518d-487d-498b-91d8-e03848d9d11d podName: nodeName:}" failed. No retries permitted until 2023-01-17 14:39:27.800848382 +0000 UTC m=+69.006440829 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-7a2b2701-6804-4389-89bd-96ed13312f78" (UniqueName: "kubernetes.io/csi/topolvm.io^2a35518d-487d-498b-91d8-e03848d9d11d") pod "edgeauthenticationserver-77667796cc-wz8px" (UID: "99254a2a-f130-4eaa-bae0-8f40af8082d8") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name topolvm.io not found in the list of registered CSI drivers

Jan 17 14:39:11 edgenius microshift[2779]: kubelet E0117 14:39:11.802407    2779 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/topolvm.io^44ce8ef5-2ca7-4cb7-aadb-45cb3fce179b podName: nodeName:}" failed. No retries permitted until 2023-01-17 14:39:27.802375101 +0000 UTC m=+69.007967648 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-a8fb267b-946e-4e11-a0ae-7e7801c4a529" (UniqueName: "kubernetes.io/csi/topolvm.io^44ce8ef5-2ca7-4cb7-aadb-45cb3fce179b") pod "edgetyperegistrydb-56b49789c7-xrvhf" (UID: "daea4ebe-2c14-4dc4-83de-a4c37d005b23") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name topolvm.io not found in the list of registered CSI drivers

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-6861 Some pods not coming up after rebooting

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is blocked by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-6861 Some pods not coming up after rebooting

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          split to

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-8021 Pods with topolvm PVC are taking a long time to start after machine reboot

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-10240 Pods with topolvm PVC are taking a long time to start after machine reboot

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-7778 Some pods are not coming up after partial cleanup of resources

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-7779 The router pod is delayed over 5m after reboot

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-10254 The router pod is delayed over 5m after reboot

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          GitHub openshift/microshift#1378: OCPBUGS-6861: Allow TopoLVM initial grace period while service-ca attempts to start

          GitHub openshift/microshift#1537: [release-4.13] OCPBUGS-10256: Allow TopoLVM initial grace period while service-ca attempts to start

          Activity

            People

              jcope@redhat.com Jon Cope
              dialvare Diego Alvarez Ponce
              John George John George
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: