Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-5223

Pod stuck in terminating if there is an LVM LogicalVolume on the underlying block PVC

XMLWordPrintable

    • None
    • 3
    • Sprint 237 - OSIntegration, Sprint 238 - OSIntegration, Sprint 239 - OSIntegration, Sprint 240 - OSIntegration, Sprint 241 - OSIntegration, Sprint 242 - OSIntegration
    • 6
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, pods in {product-title} that access block PVC storage with LVM metadata could get stuck when terminating. This is because the same LVM devices were active both inside the container and on the host. An example of this occurred when running a virtual machine inside a pod using {VirtProductName} that uses LVM for the virtual machine. With this update, {op-system} by default only attempts to setup and access devices that are in the `/etc/lvm/devices/system.devices` file. This prevents contentious access to the LVM devices inside the virtual machine guests. (link:https://issues.redhat.com/browse/OCPBUGS-5223[*OCPBUGS-5223*])
      Show
      Previously, pods in {product-title} that access block PVC storage with LVM metadata could get stuck when terminating. This is because the same LVM devices were active both inside the container and on the host. An example of this occurred when running a virtual machine inside a pod using {VirtProductName} that uses LVM for the virtual machine. With this update, {op-system} by default only attempts to setup and access devices that are in the `/etc/lvm/devices/system.devices` file. This prevents contentious access to the LVM devices inside the virtual machine guests. (link: https://issues.redhat.com/browse/OCPBUGS-5223 [* OCPBUGS-5223 *])
    • Bug Fix
    • Done

      Description of problem:

      Kubelet cannot finish terminating a pod that uses a PVC with volumeMode: Block if the there is an LVM LogicalVolume on the underlying block device

      Version-Release number of selected component (if applicable):

      OCP 4.12

      How reproducible:

      100%

      Steps to Reproduce:

      1. Create reproducer manifest below
2. Edit /etc/lvm/lvm.conf, make sure to uncomment
types = [ "fd", 16, "rbd", 16 ]
3. 
pvcreate /dev/myvol
vgcreate vol /dev/myvol
lvcreate -L 1G -n test vol
4. Kill pod

      Actual results:

      Stuck in terminating

      Expected results:

      Cleans up well

      Additional info:

      This happens (thx to Nijin Ashok)
because although the LV was created from the pod, the loop device is 
directly linked to rbd device and hence it will be discoverable on the node level.
This discovery will lead to creating a dm device on top of the loop device, which will result in the kubelet failing to umount:
umount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev/ae3b79e4-e551-4338-a8b4-f2f18e8c1469: target is busy.

One W/A is LVM filters which will denylist all the loop devices from the eyes of the node, this will prevent the activation of the pod LVs in the node.

I would understand the argument that this use case is less interesting
with plain k8s, but we actually hit this in OpenShift Virtualization where 
people created LVs in their VMs.
There is a dusty issue about this on kubernetes/kubernetes:
https://github.com/kubernetes/kubernetes/issues/110163

$ cat reproducer.yaml
*struggling to reproduce with pods*

       

              rhn-gps-dmabe Dusty Mabe
              akalenyu Alex Kalenyuk
              Michael Nguyen Michael Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

                Created:
                Updated:
                Resolved: