Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30950

ovnkube-node hostPath mount of /var/lib/kubelet is missing HostToContainer mountPropagation, breaks CSI driver

XMLWordPrintable

    • Important
    • Yes
    • OCPNODE Sprint 252 (Blue)
    • 1
    • Hide

      None

      Show
      None
    • 05/07; Workaround is to restart ovnkube-node pods regularly or patch the ovnkube-node ds volume mount; priority seems too low to me; this is a genuine bug; cu says "Business impact is High. Production Workload cannot start after Pods are scheduled."

      Description of problem: ovnkube-node and multus DaemonSets have hostPath volumes which prevent clean unmount of CSI Volumes because of missing "mountPropagation: HostToContainer" parameter in volumeMount

      Version-Release number of selected component (if applicable):  OpenShift 4.14

      How reproducible:  Always

      Steps to Reproduce:

      1. on a node mount a file system underneath /var/lib/kubelet/ simulating the mount of a  CSI driver PersistentVolume

      2. restart the ovnkube-node pod running on that node

      3. unmount the filesystem from 1. The mount will then be removed from the host list of mounted devices however a copy of the mount is still active in the mount namespace of the ovnkube-node pod.
      This is blocking some CSI drivers relying on multipath to properly delete a block device, since mounts are still registered on the block device.
       

      Actual results:
      CSI Volume Mount cleanly unmounted.
       

      Expected results:
      CSI Volume Mount uncleanly unmounted.
       

      Additional info:

      The mountPropagation parameter is already implememted in the volumeMount for the host rootFS:

                  - name: host-slash
                    readOnly: true
                    mountPath: /host
                    mountPropagation: HostToContainer

       However the same parameter is missing for the volumeMount of /var/lib/kubelet

      It is possible to workaround the issue with a kubectl patch command like this:

      $ kubectl patch daemonset ovnkube-node --type='json' -p='[
        {
          "op": "replace",
          "path": "/spec/template/spec/containers/7/volumeMounts/1",
          "value": {
            "name": "host-kubelet",
            "mountPath": "/var/lib/kubelet",
            "mountPropagation": "HostToContainer",
            "readOnly": true
         }
       }
      ]'

       

      Affected Platforms: Platform Agnostic UPI

            rhn-support-arghosh Arnab Ghosh
            rhn-support-ekasprzy Emmanuel Kasprzyk
            Martin Kennelly
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: