Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-9308

node-volume-attach-limit not considered

XMLWordPrintable

    • Quality / Stability / Reliability
    • None
    • None
    • None
    • Low
    • None
    • Unspecified
    • None
    • None
    • Rejected
    • None
    • ?
    • None
    • Known Issue
    • Hide
      Cause: CSINode's count isn't updated by kubelet when CSIDriver provides a new value.

      Consequence: This causes node-volume-attach-limit cloud provider option modifications to be ignored.

      Workaround (if any): After modification delete CSINode objects and restart kubelet on all the nodes. You might also just delete all CSINode objects before the modification of cloud provider config.

      Result:
      Show
      Cause: CSINode's count isn't updated by kubelet when CSIDriver provides a new value. Consequence: This causes node-volume-attach-limit cloud provider option modifications to be ignored. Workaround (if any): After modification delete CSINode objects and restart kubelet on all the nodes. You might also just delete all CSINode objects before the modification of cloud provider config. Result:
    • None
    • None
    • None
    • None

      Description of problem:

      Thanks to the work included on https://bugzilla.redhat.com/show_bug.cgi?id=2065597, now it is possible to configure cinder-csi by editing cloud-provider-config.

      One of the parameters included is 'node-volume-attach-limit' [1]. Changing the param to 10 and running 11 pods connected to 11 different PVCs in the same worker is accepted:

      $ oc get cm -n openshift-cluster-csi-drivers cloud-conf -o yaml
      apiVersion: v1
      data:
      cloud.conf: |+
      [Global]
      region = regionOne
      ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
      use-clouds = true
      clouds-file = /etc/kubernetes/secret/clouds.yaml
      cloud = openstack

      [LoadBalancer]
      use-octavia = True

      [BlockStorage]
      ignore-volume-az = yes
      node-volume-attach-limit = 10

      [Metadata]
      search-order = metadataService

      $ oc get pods
      NAME READY STATUS RESTARTS AGE
      demo-0-8466fc476-mbll5 1/1 Running 0 2m17s
      demo-1-649948d68-dnhsv 1/1 Running 0 2m14s
      demo-10-676c554976-52bwt 1/1 Running 0 103s
      demo-2-6947d8d46d-mtxn6 1/1 Running 0 2m11s
      demo-3-79db5fd4d8-qxgbl 1/1 Running 0 2m7s
      demo-4-65678d5484-qlngk 1/1 Running 0 2m4s
      demo-5-f75648d76-wzcmw 1/1 Running 0 2m
      demo-6-856dc89dd8-s9bp4 1/1 Running 0 117s
      demo-7-64f5944f8d-shq4d 1/1 Running 0 114s
      demo-8-7b677bb644-4bhnn 1/1 Running 0 111s
      demo-9-9b88b4c56-m7swp 1/1 Running 0 107s
      $ oc get pvc
      NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
      pvc-0 Bound pvc-92aaeab8-6c60-4f32-9ab6-a703761ad3cd 1Gi RWO topology-aware-0 2m19s
      pvc-1 Bound pvc-336915af-8154-4e67-b37d-2fa93f2db645 1Gi RWO topology-aware-1 2m16s
      pvc-10 Bound pvc-a7f58825-e465-4e2b-b9ec-36362589d1d2 1Gi RWO topology-aware-10 105s
      pvc-2 Bound pvc-4ded3dab-170d-4573-ba29-8bb10b3e857b 1Gi RWO topology-aware-2 2m13s
      pvc-3 Bound pvc-b3d7ec16-5b7e-4964-9a0a-29480dee31a1 1Gi RWO topology-aware-3 2m9s
      pvc-4 Bound pvc-a6ae7415-00b5-4849-8065-7fe7acace165 1Gi RWO topology-aware-4 2m6s
      pvc-5 Bound pvc-246421b3-2716-41c8-88e9-d6253c49acdd 1Gi RWO topology-aware-5 2m2s
      pvc-6 Bound pvc-61ec111a-d5ae-47d2-83be-1a4a344afea7 1Gi RWO topology-aware-6 119s
      pvc-7 Bound pvc-b65cd362-977a-4613-84a3-bb75b1adf971 1Gi RWO topology-aware-7 116s
      pvc-8 Bound pvc-a79d7bb1-9697-4c1b-8ad3-696a0c2edafd 1Gi RWO topology-aware-8 113s
      pvc-9 Bound pvc-48e37bc7-ff50-4517-8cac-a57fd6f0f47b 1Gi RWO topology-aware-9 109s
      $ openstack volume list -c 'Attached to' | grep ostest | awk -F' ' '

      {print $4}

      ' | sort | uniq -c
      11 ostest-zh8g7-worker-0-7gl4r
      $ openstack volume list
      --------------------------------------------------------------------------------------------------------------------------------------------

      ID Name Status Size Attached to

      --------------------------------------------------------------------------------------------------------------------------------------------

      ce816f7d-6873-457a-acfa-327cf9357c4a pvc-a7f58825-e465-4e2b-b9ec-36362589d1d2 in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdl
      dd774cc3-488c-4fa7-bf5d-f84c3d67f99b pvc-48e37bc7-ff50-4517-8cac-a57fd6f0f47b in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdk
      1fc32fa8-feeb-4f06-a712-5b9da301e187 pvc-a79d7bb1-9697-4c1b-8ad3-696a0c2edafd in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdj
      0ced70af-0a85-4cd2-bfab-f7299b4bd128 pvc-b65cd362-977a-4613-84a3-bb75b1adf971 in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdi
      dc413629-382b-492b-9e41-ae9163ff542f pvc-61ec111a-d5ae-47d2-83be-1a4a344afea7 in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdh
      7aab9069-dc48-4413-b4b9-f7c31bdaeaa6 pvc-246421b3-2716-41c8-88e9-d6253c49acdd in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdg
      c1656f81-3d55-4f15-aab8-d84b37ceae9b pvc-a6ae7415-00b5-4849-8065-7fe7acace165 in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdf
      d8608274-5a75-4daf-adb5-575160b296e3 pvc-b3d7ec16-5b7e-4964-9a0a-29480dee31a1 in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vde
      c59bc638-f410-4d3e-a3b9-d31288a0c4a0 pvc-4ded3dab-170d-4573-ba29-8bb10b3e857b in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdd
      7f779063-61f1-462b-bb9d-f77429dc9485 pvc-336915af-8154-4e67-b37d-2fa93f2db645 in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdc
      2c17fa64-1b3a-44e1-8ad0-8fb5c4e3ffc1 pvc-92aaeab8-6c60-4f32-9ab6-a703761ad3cd in-use 1 Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdb

      --------------------------------------------------------------------------------------------------------------------------------------------

      [1] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md#driver-config

      Version-Release number of selected component (if applicable): 4.11.0-0.nightly-2022-06-06-025509 RHOS-16.2-RHEL-8-20220311.n.1

      How reproducible: Always

      Steps to Reproduce:
      1. Change cloud cm to node-volume-attach-limit = 10
      2. Wait until cluster re-configuration is completed (around 30 minutes)
      3. Create 11 pods using cinder PVCs with NodeSelector set to use always the same worker.

      Actual results: The node-volume-attach-limit is ignored.
      Expected results: The PVC number 11 should not becreated.

              rhn-gps-mbooth Matthew Booth
              rlobillo Ramón Lobillo
              None
              None
              Jon Uriarte Jon Uriarte
              None
              Red Hat Employee
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: