Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-3774

Unable to use application credentials for Cinder CSI after OpenStack credentials update

XMLWordPrintable

    • +
    • Important
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously when {rh-openstack-first credentials} were rotated, an issue occurred where the {rh-openstack} Cinder Container Storage Interface (CSI) driver would use old credentials until the driver was restarted. If the old credentials expired then all volume operations would fail.

      For the {product-title} {product-version} fixes this issue by automatically updating the {rh-openstack} Cinder CSI driver when credentials are rotated.

      (link:https://issues.redhat.com/browse/OCPBUGS-3774[*OCPBUGS-3774*])
      Show
      Previously when {rh-openstack-first credentials} were rotated, an issue occurred where the {rh-openstack} Cinder Container Storage Interface (CSI) driver would use old credentials until the driver was restarted. If the old credentials expired then all volume operations would fail. For the {product-title} {product-version} fixes this issue by automatically updating the {rh-openstack} Cinder CSI driver when credentials are rotated. (link: https://issues.redhat.com/browse/OCPBUGS-3774 [* OCPBUGS-3774 *])
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-2845. The following is the description of the original issue:

      Description of problem:

      After OpenStack application credentials update, Cinder CSI tries to use the previous credentials and fails with an "Unauthorized" 401 code:
      
      MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}
      

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-10-25-07574 on top of RHOS-16.2-RHEL-8-20220804.n.1
      

      How reproducible:

      Always and with all of the network types

      Steps to Reproduce:

      1. Install OCP 4.12 with application credentials on top of OSP:
      $ source shiftstackrc && openstack application credential list
      +----------------------------------+--------------------+----------------------------------+-----------------------+------------+
      | ID                               | Name               | Project ID                       | Description           | Expires At |
      +----------------------------------+--------------------+----------------------------------+-----------------------+------------+
      | 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles | None       |
      +----------------------------------+--------------------+----------------------------------+-----------------------+------------+
      
      $ grep -A 10 shiftstack clouds.yaml
      #BEGIN shiftstack PARAMETERS
       shiftstack:
          auth:
              application_credential_id: 339b768af8a44b1a95a5b30872a1a492
              application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw
              auth_url: https://10.46.44.140:13000
          auth_type: v3applicationcredential
          cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem
          identity_api_version: '3'
          region_name: regionOne
      #END shiftstack PARAMETERS
      
      $ oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.12.0-0.nightly-2022-10-25-075747   True        False         26m     Cluster version is 4.12.0-0.nightly-2022-10-25-075747
      
      
      2. Deploy a namespace with PVC and a cinder.csi.openstack.org StorageClass:
      $ oc project topologyaware-test
      Now using project "topologyaware-test" on server "https://api.ostest.shiftstack.com:6443".
      (shiftstack) [stack@undercloud-0 ~]$ oc get pods
      NAME                      READY   STATUS    RESTARTS   AGE
      demo-0-6cd78b9fdc-w6xf6   1/1     Running   0          4m15s
      demo-1-fdc6c877c-5pcpp    1/1     Running   0          4m8s
      demo-2-f8c498bbc-pdl4r    1/1     Running   0          4m2s
      $ oc get sc
      NAME                     PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
      csi-manila-default       manila.csi.openstack.org   Delete          Immediate              false                  59m
      standard-csi (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   59m
      topology-aware-0         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  5m33s
      topology-aware-1         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  5m16s
      topology-aware-2         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  4m55s
      $ oc get pvc
      NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
      pvc-0   Bound    pvc-12eede08-ebb0-4ced-8d1e-3aa8dc6ec7c9   1Gi        RWO            topology-aware-0   5m40s
      pvc-1   Bound    pvc-0ed7386f-8962-4975-ad28-21ac8dcacb0b   1Gi        RWO            topology-aware-1   5m23s
      pvc-2   Bound    pvc-5f50bb91-1b8e-467f-8a06-577d7750bda2   1Gi        RWO            topology-aware-2   5m2s
      
      
      3. Update OpenStack application credentials [0]
      3.1. Create a new application credential:
      $ openstack application credential create --description "App Creds - All roles - New" AppCredsShiftstackNew
      +--------------+----------------------------------------------------------------------------------------+
      | Field        | Value                                                                                  |
      +--------------+----------------------------------------------------------------------------------------+
      | description  | App Creds - All roles - New                                                            |
      | expires_at   | None                                                                                   |
      | id           | 534373a90fde41f39da826a5d77a6441                                                       |
      | name         | AppCredsShiftstackNew                                                                  |
      | project_id   | 2073ea9b31cd4a17a72046da0f6e002c                                                       |
      | roles        | swiftoperator member reader                                                            |
      | secret       | _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA |
      | system       | None                                                                                   |
      | unrestricted | False                                                                                  |
      | user_id      | 67ac336163bb4606ac3ade19454c9089                                                       |
      +--------------+----------------------------------------------------------------------------------------+
      
      $ openstack application credential list
      +----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
      | ID                               | Name                  | Project ID                       | Description                 | Expires At |
      +----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
      | 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack    | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles       | None       |
      | 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None       |
      +----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
      
      3.2. Update the clouds.yaml with the openstack cloud and the new credential:
      $ vim clouds.yaml
      $ diff clouds.yaml clouds.yaml.bck
      28c28
      <  openstack:
      ---
      >  shiftstack:
      30,31c30,31
      <         application_credential_id: 534373a90fde41f39da826a5d77a6441
      <         application_credential_secret: _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA
      ---
      >         application_credential_id: 339b768af8a44b1a95a5b30872a1a492
      >         application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw
      
      3.3. Upload the new clouds.yaml to the openstack-credentials secret in the kube-system namespace:
      oc set data -n kube-system secret/openstack-credentials clouds.yaml="$(<~/clouds.yaml)"
      
      3.4. Make sure the cred updated:
      $ oc get secret -n kube-system openstack-credentials -o json | jq -r '.data."clouds.yaml"' | base64 -d
      
      
      4. At this point operations with cinder csi are working fine.
      Delete the project which includes cinder csi PVCs:
      $ oc delete project topologyaware-test
      
      
      5. Remove from openstack the old app cred:
      $ openstack application credential list                                                                                                                                        
      +----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
      | ID                               | Name                  | Project ID                       | Description                 | Expires At |
      +----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
      | 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack    | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles       | None       |
      | 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None       |                                                                        
      +----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+                                                                        
      (shiftstack) [stack@undercloud-0 ~]$ openstack application credential delete 339b768af8a44b1a95a5b30872a1a492
      
      
      6. Apply a deployment with pvc: [1]
      $ oc new-project topologyaware-test
      $ oc apply -f cinder_csi_0.yaml
      storageclass.storage.k8s.io/topology-aware-0 unchanged
      persistentvolumeclaim/pvc-0 created
      deployment.apps/demo-0 created
      
      
      7. The kubelet tries to use the previous credentials and fails with an "Unauthorized" 401 code, and the pod stuck on ContainerCreating:
      $ oc get pods
      NAME                      READY   STATUS              RESTARTS   AGE
      demo-0-6cd78b9fdc-2glz5   0/1     ContainerCreating   0          20m
      
      $ oc describe pod demo-0-6cd78b9fdc-2glz5 | grep -A 20 Events
      Events:
        Type     Reason                  Age                  From                     Message
        ----     ------                  ----                 ----                     -------
        Normal   Scheduled               24m                  default-scheduler        Successfully assigned topologyaware-test/demo-0-6cd78b9fdc-2glz5 to ostest-pcw46-worker-0-s9zxj
        Normal   SuccessfulAttachVolume  24m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5"
        Warning  FailedMount             6m25s (x6 over 22m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[mydata kube-api-access-9t6h4]: timed out waiting for the condition
        Warning  FailedMount             111s (x4 over 17m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[kube-api-access-9t6h4 mydata]: timed out waiting for the condition
        Warning  FailedMount             100s (x19 over 24m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead
      {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}
      

      [0] https://github.com/shiftstack/installer/blob/master/docs/user/openstack/README.md#openstack-credentials-update

      [1] [^cinder_csi_0.yaml]

      Actual results:

      Unable to use application credentials for cinder-csi after OpenStack credentials update

      Expected results:

      Successful to use application credentials for cinder-csi after OpenStack credentials update

      Additional info:

      The 401 error is also present on the openshift-cloud-controller-manager:
      $ oc logs -n openshift-cloud-controller-manager openstack-cloud-controller-manager-6f6f9c8979-tzfqz | grep 401 | tail -1
      {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}

      Attached [^must-gather.tar.gz]

       

            rhn-gps-mbooth Matthew Booth
            openshift-crt-jira-prow OpenShift Prow Bot
            Itshak Brown Itshak Brown
            Darragh Fitzmaurice Darragh Fitzmaurice
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: