Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30266

LVMS: FailedMount a small (10Mi) Filesystem PVC

XMLWordPrintable

    • Moderate
    • No
    • 5
    • OCPEDGE Sprint 250, OCPEDGE Sprint 251, OCPEDGE Sprint 252
    • 3
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, LVM Storage did not support minimum storage size for persistent volume claims (PVCs). This can lead to mount failures while provisioning PVCs. With this release, LVM Storage supports minimum storage size for PVCs. The following are the minimum storage sizes that you can request for each file system type:

      - block: 8 MiB

      - xfs: 300 MiB

      - ext4: 32 MiB

      If the value of the requests.storage field in the PersistentVolumeClaim object is less than the minimum storage size, the requested storage size is rounded to the minimum storage size. If the value of the limits.storage field is less than the minimum storage size, PVC creation fails with an error.
      Show
      Previously, LVM Storage did not support minimum storage size for persistent volume claims (PVCs). This can lead to mount failures while provisioning PVCs. With this release, LVM Storage supports minimum storage size for PVCs. The following are the minimum storage sizes that you can request for each file system type: - block: 8 MiB - xfs: 300 MiB - ext4: 32 MiB If the value of the requests.storage field in the PersistentVolumeClaim object is less than the minimum storage size, the requested storage size is rounded to the minimum storage size. If the value of the limits.storage field is less than the minimum storage size, PVC creation fails with an error.
    • Bug Fix
    • Done

      Description of problem:

      Pod fails to mount a small (10Mi) Filesystem PVC
      
          output:(size 3072 of data subvolume is too small, minimum 4096 blocks

      Version-Release number of selected component (if applicable):

      4.15

      Steps to Reproduce:

      1. Create a PVC:
      
      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: pvc-lvms-ten-mi-fs
      spec:
        volumeMode: Filesystem
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Mi
        storageClassName: lvms-vg1
      
      ------
      $ oc get pvc
      NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      pvc-lvms-ten-mi-fs   Bound    pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a   10Mi       RWO            lvms-vg1       107s

       

      2. Create a pod:
      
      apiVersion: v1
      kind: Pod
      metadata:
        name: test-pod
      spec:
        containers:
          - command:
              - sleep
              - "3600000"
            image: "quay.io/openshift-cnv/qe-cnv-tests-net-util-container:centos-stream-9"
            imagePullPolicy: IfNotPresent
            name: test-pod      
            volumeMounts:
              - mountPath: /foo
                name: data-vol
        volumes:
          - name: data-vol
            persistentVolumeClaim:
              claimName: pvc-lvms-ten-mi-fs

       

      Actual results:

      Pod is ContainerCreating
      
      $ oc get pods
      NAME       READY   STATUS              RESTARTS   AGE
      test-pod   0/1     ContainerCreating   0          90s

      In pod's events:

      $ oc describe pod test-pod
      ....
        Warning  FailedMount  5s (x5 over 13s)  kubelet            MountVolume.SetUp failed for volume "pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a" : rpc error: code = Internal desc = mount failed: volume=17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5, error=format of disk "/dev/topolvm/17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5" failed: type:("xfs") target:("/var/lib/kubelet/pods/8b4ae8e7-55c0-45d5-805b-02496d66b782/volumes/kubernetes.io~csi/pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a/mount") options:("nouuid,defaults") errcode:(exit status 1) output:(size 3072 of data subvolume is too small, minimum 4096 blocks

       

      Expected results:

      Pod could mount a PVC. 
      If provisioner can't handle a small PVC, it should provison a larger PVC.

      Additional info:

      Full pod describe:
      
      $ oc describe pod test-pod
      Name:             test-pod
      Namespace:        default
      Priority:         0
      Service Account:  default
      Node:             cnv-qe-infra-23.cnvqe2.lab.eng.rdu2.redhat.com/10.1.156.23
      Start Time:       Tue, 05 Mar 2024 16:37:22 +0000
      Labels:           <none>
      Annotations:      k8s.ovn.org/pod-networks:
                          {"default":{"ip_addresses":["10.128.0.95/23"],"mac_address":"0a:58:0a:80:00:5f","gateway_ips":["10.128.0.1"],"routes":[{"dest":"10.128.0.0...
      Status:           Pending
      IP:               
      IPs:              <none>
      Containers:
        test-pod:
          Container ID:  
          Image:         quay.io/openshift-cnv/qe-cnv-tests-net-util-container:centos-stream-9
          Image ID:      
          Port:          <none>
          Host Port:     <none>
          Command:
            sleep
            3600000
          State:          Waiting
            Reason:       ContainerCreating
          Ready:          False
          Restart Count:  0
          Environment:    <none>
          Mounts:
            /foo from data-vol (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qmcrp (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        data-vol:
          Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
          ClaimName:  pvc-lvms-ten-mi-fs
          ReadOnly:   false
        kube-api-access-qmcrp:
          Type:                    Projected (a volume that contains injected data from multiple sources)
          TokenExpirationSeconds:  3607
          ConfigMapName:           kube-root-ca.crt
          ConfigMapOptional:       <nil>
          DownwardAPI:             true
          ConfigMapName:           openshift-service-ca.crt
          ConfigMapOptional:       <nil>
      QoS Class:                   BestEffort
      Node-Selectors:              <none>
      Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
      Events:
        Type     Reason       Age                 From               Message
        ----     ------       ----                ----               -------
        Normal   Scheduled    2m4s                default-scheduler  Successfully assigned default/test-pod to cnv-qe-infra-23.cnvqe2.lab.eng.rdu2.redhat.com
        Warning  FailedMount  59s (x8 over 2m4s)  kubelet            MountVolume.SetUp failed for volume "pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a" : rpc error: code = Internal desc = mount failed: volume=17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5, error=format of disk "/dev/topolvm/17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5" failed: type:("xfs") target:("/var/lib/kubelet/pods/8b4ae8e7-55c0-45d5-805b-02496d66b782/volumes/kubernetes.io~csi/pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a/mount") options:("nouuid,defaults") errcode:(exit status 1) output:(size 3072 of data subvolume is too small, minimum 4096 blocks
      Usage: mkfs.xfs
      /* blocksize */          [-b size=num]
      /* config file */      [-c options=xxx]
      /* metadata */           [-m crc=0|1,finobt=0|1,uuid=xxx,rmapbt=0|1,reflink=0|1,
                                     inobtcount=0|1,bigtime=0|1]
      /* data subvol */      [-d agcount=n,agsize=n,file,name=xxx,size=num,
                                     (sunit=value,swidth=value|su=num,sw=num|noalign),
                                     sectsize=num
      /* force overwrite */  [-f]
      /* inode size */       [-i perblock=n|size=num,maxpct=n,attr=0|1|2,
                                     projid32bit=0|1,sparse=0|1,nrext64=0|1]
      /* no discard */       [-K]
      /* log subvol */       [-l agnum=n,internal,size=num,logdev=xxx,version=n
                                     sunit=value|su=num,sectsize=num,lazy-count=0|1]
      /* label */              [-L label (maximum 12 characters)]
      /* naming */             [-n size=num,version=2|ci,ftype=0|1]
      /* no-op info only */  [-N]
      /* prototype file */   [-p fname]
      /* quiet */              [-q]
      /* realtime subvol */  [-r extsize=num,size=num,rtdev=xxx]
      /* sectorsize */       [-s size=num]
      /* version */            [-V]
                                 devicename
      <devicename> is required unless -d name=xxx is given.
      <num> is xxx (bytes), xxxs (sectors), xxxb (fs blocks), xxxk (xxx KiB),
            xxxm (xxx MiB), xxxg (xxx GiB), xxxt (xxx TiB) or xxxp (xxx PiB).
      <value> is xxx (512 byte blocks).
      )
      

            rh-ee-jmoller Jakob Moeller
            jpeimer@redhat.com Jenia Peimer
            Rahul Deore Rahul Deore
            Srikanth R Srikanth R
            Alex Kalenyuk
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: