Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32160

[release-4.15] LVMS: FailedMount a small (10Mi) Filesystem PVC

XMLWordPrintable

    • Moderate
    • No
    • 2
    • OCPEDGE Sprint 250, OCPEDGE Sprint 251, OCPEDGE Sprint 252
    • 3
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, there was no minimum size for Volumes which could lead to edge cases in which the logical volume metadata or filesystem metadata might not fit into the given volume size. To accomodate this, new minimum PVC sizes were introduced that will be defaulted to if no limit is set and the request is below these minimum requested sizes. If there is a limit that is lower than these minimum sizes, an error is now produced.

      The minimum provisioning size is as follows:
      XFS: 300Mi
      ext4: 32Mi
      Block Storage: 8Mi
      Show
      Previously, there was no minimum size for Volumes which could lead to edge cases in which the logical volume metadata or filesystem metadata might not fit into the given volume size. To accomodate this, new minimum PVC sizes were introduced that will be defaulted to if no limit is set and the request is below these minimum requested sizes. If there is a limit that is lower than these minimum sizes, an error is now produced. The minimum provisioning size is as follows: XFS: 300Mi ext4: 32Mi Block Storage: 8Mi
    • Bug Fix
    • In Progress

      Description of problem:

      Pod fails to mount a small (10Mi) Filesystem PVC
      
          output:(size 3072 of data subvolume is too small, minimum 4096 blocks

      Version-Release number of selected component (if applicable):

      4.15

      Steps to Reproduce:

      1. Create a PVC:
      
      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: pvc-lvms-ten-mi-fs
      spec:
        volumeMode: Filesystem
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Mi
        storageClassName: lvms-vg1
      
      ------
      $ oc get pvc
      NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      pvc-lvms-ten-mi-fs   Bound    pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a   10Mi       RWO            lvms-vg1       107s

       

      2. Create a pod:
      
      apiVersion: v1
      kind: Pod
      metadata:
        name: test-pod
      spec:
        containers:
          - command:
              - sleep
              - "3600000"
            image: "quay.io/openshift-cnv/qe-cnv-tests-net-util-container:centos-stream-9"
            imagePullPolicy: IfNotPresent
            name: test-pod      
            volumeMounts:
              - mountPath: /foo
                name: data-vol
        volumes:
          - name: data-vol
            persistentVolumeClaim:
              claimName: pvc-lvms-ten-mi-fs

       

      Actual results:

      Pod is ContainerCreating
      
      $ oc get pods
      NAME       READY   STATUS              RESTARTS   AGE
      test-pod   0/1     ContainerCreating   0          90s

      In pod's events:

      $ oc describe pod test-pod
      ....
        Warning  FailedMount  5s (x5 over 13s)  kubelet            MountVolume.SetUp failed for volume "pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a" : rpc error: code = Internal desc = mount failed: volume=17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5, error=format of disk "/dev/topolvm/17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5" failed: type:("xfs") target:("/var/lib/kubelet/pods/8b4ae8e7-55c0-45d5-805b-02496d66b782/volumes/kubernetes.io~csi/pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a/mount") options:("nouuid,defaults") errcode:(exit status 1) output:(size 3072 of data subvolume is too small, minimum 4096 blocks

       

      Expected results:

      Pod could mount a PVC. 
      If provisioner can't handle a small PVC, it should provison a larger PVC.

      Additional info:

      Full pod describe:
      
      $ oc describe pod test-pod
      Name:             test-pod
      Namespace:        default
      Priority:         0
      Service Account:  default
      Node:             cnv-qe-infra-23.cnvqe2.lab.eng.rdu2.redhat.com/10.1.156.23
      Start Time:       Tue, 05 Mar 2024 16:37:22 +0000
      Labels:           <none>
      Annotations:      k8s.ovn.org/pod-networks:
                          {"default":{"ip_addresses":["10.128.0.95/23"],"mac_address":"0a:58:0a:80:00:5f","gateway_ips":["10.128.0.1"],"routes":[{"dest":"10.128.0.0...
      Status:           Pending
      IP:               
      IPs:              <none>
      Containers:
        test-pod:
          Container ID:  
          Image:         quay.io/openshift-cnv/qe-cnv-tests-net-util-container:centos-stream-9
          Image ID:      
          Port:          <none>
          Host Port:     <none>
          Command:
            sleep
            3600000
          State:          Waiting
            Reason:       ContainerCreating
          Ready:          False
          Restart Count:  0
          Environment:    <none>
          Mounts:
            /foo from data-vol (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qmcrp (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        data-vol:
          Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
          ClaimName:  pvc-lvms-ten-mi-fs
          ReadOnly:   false
        kube-api-access-qmcrp:
          Type:                    Projected (a volume that contains injected data from multiple sources)
          TokenExpirationSeconds:  3607
          ConfigMapName:           kube-root-ca.crt
          ConfigMapOptional:       <nil>
          DownwardAPI:             true
          ConfigMapName:           openshift-service-ca.crt
          ConfigMapOptional:       <nil>
      QoS Class:                   BestEffort
      Node-Selectors:              <none>
      Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
      Events:
        Type     Reason       Age                 From               Message
        ----     ------       ----                ----               -------
        Normal   Scheduled    2m4s                default-scheduler  Successfully assigned default/test-pod to cnv-qe-infra-23.cnvqe2.lab.eng.rdu2.redhat.com
        Warning  FailedMount  59s (x8 over 2m4s)  kubelet            MountVolume.SetUp failed for volume "pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a" : rpc error: code = Internal desc = mount failed: volume=17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5, error=format of disk "/dev/topolvm/17c48e86-3a9b-4ef0-83c1-698e0d6fd7b5" failed: type:("xfs") target:("/var/lib/kubelet/pods/8b4ae8e7-55c0-45d5-805b-02496d66b782/volumes/kubernetes.io~csi/pvc-1bba5d94-aa42-4f8e-a75e-fe16debdbd9a/mount") options:("nouuid,defaults") errcode:(exit status 1) output:(size 3072 of data subvolume is too small, minimum 4096 blocks
      Usage: mkfs.xfs
      /* blocksize */          [-b size=num]
      /* config file */      [-c options=xxx]
      /* metadata */           [-m crc=0|1,finobt=0|1,uuid=xxx,rmapbt=0|1,reflink=0|1,
                                     inobtcount=0|1,bigtime=0|1]
      /* data subvol */      [-d agcount=n,agsize=n,file,name=xxx,size=num,
                                     (sunit=value,swidth=value|su=num,sw=num|noalign),
                                     sectsize=num
      /* force overwrite */  [-f]
      /* inode size */       [-i perblock=n|size=num,maxpct=n,attr=0|1|2,
                                     projid32bit=0|1,sparse=0|1,nrext64=0|1]
      /* no discard */       [-K]
      /* log subvol */       [-l agnum=n,internal,size=num,logdev=xxx,version=n
                                     sunit=value|su=num,sectsize=num,lazy-count=0|1]
      /* label */              [-L label (maximum 12 characters)]
      /* naming */             [-n size=num,version=2|ci,ftype=0|1]
      /* no-op info only */  [-N]
      /* prototype file */   [-p fname]
      /* quiet */              [-q]
      /* realtime subvol */  [-r extsize=num,size=num,rtdev=xxx]
      /* sectorsize */       [-s size=num]
      /* version */            [-V]
                                 devicename
      <devicename> is required unless -d name=xxx is given.
      <num> is xxx (bytes), xxxs (sectors), xxxb (fs blocks), xxxk (xxx KiB),
            xxxm (xxx MiB), xxxg (xxx GiB), xxxt (xxx TiB) or xxxp (xxx PiB).
      <value> is xxx (512 byte blocks).
      )
      

              rh-ee-jmoller Jakob Moeller (Inactive)
              jpeimer@redhat.com Jenia Peimer
              Rahul Deore Rahul Deore
              Srikanth R Srikanth R
              Alex Kalenyuk
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: