Unable to use application credentials for Cinder CSI after OpenStack credentials update

After OpenStack application credentials update, Cinder CSI tries to use the previous credentials and fails with an "Unauthorized" 401 code:

MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}

4.12.0-0.nightly-2022-10-25-07574 on top of RHOS-16.2-RHEL-8-20220804.n.1

Always and with all of the network types

1. Install OCP 4.12 with application credentials on top of OSP:
$ source shiftstackrc && openstack application credential list
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| ID                               | Name               | Project ID                       | Description           | Expires At |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles | None       |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+

$ grep -A 10 shiftstack clouds.yaml
#BEGIN shiftstack PARAMETERS
 shiftstack:
    auth:
        application_credential_id: 339b768af8a44b1a95a5b30872a1a492
        application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw
        auth_url: https://10.46.44.140:13000
    auth_type: v3applicationcredential
    cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem
    identity_api_version: '3'
    region_name: regionOne
#END shiftstack PARAMETERS

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-10-25-075747   True        False         26m     Cluster version is 4.12.0-0.nightly-2022-10-25-075747


2. Deploy a namespace with PVC and a cinder.csi.openstack.org StorageClass:
$ oc project topologyaware-test
Now using project "topologyaware-test" on server "https://api.ostest.shiftstack.com:6443".
(shiftstack) [stack@undercloud-0 ~]$ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
demo-0-6cd78b9fdc-w6xf6   1/1     Running   0          4m15s
demo-1-fdc6c877c-5pcpp    1/1     Running   0          4m8s
demo-2-f8c498bbc-pdl4r    1/1     Running   0          4m2s
$ oc get sc
NAME                     PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-default       manila.csi.openstack.org   Delete          Immediate              false                  59m
standard-csi (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   59m
topology-aware-0         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  5m33s
topology-aware-1         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  5m16s
topology-aware-2         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  4m55s
$ oc get pvc
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
pvc-0   Bound    pvc-12eede08-ebb0-4ced-8d1e-3aa8dc6ec7c9   1Gi        RWO            topology-aware-0   5m40s
pvc-1   Bound    pvc-0ed7386f-8962-4975-ad28-21ac8dcacb0b   1Gi        RWO            topology-aware-1   5m23s
pvc-2   Bound    pvc-5f50bb91-1b8e-467f-8a06-577d7750bda2   1Gi        RWO            topology-aware-2   5m2s


3. Update OpenStack application credentials [0]
3.1. Create a new application credential:
$ openstack application credential create --description "App Creds - All roles - New" AppCredsShiftstackNew
+--------------+----------------------------------------------------------------------------------------+
| Field        | Value                                                                                  |
+--------------+----------------------------------------------------------------------------------------+
| description  | App Creds - All roles - New                                                            |
| expires_at   | None                                                                                   |
| id           | 534373a90fde41f39da826a5d77a6441                                                       |
| name         | AppCredsShiftstackNew                                                                  |
| project_id   | 2073ea9b31cd4a17a72046da0f6e002c                                                       |
| roles        | swiftoperator member reader                                                            |
| secret       | _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA |
| system       | None                                                                                   |
| unrestricted | False                                                                                  |
| user_id      | 67ac336163bb4606ac3ade19454c9089                                                       |
+--------------+----------------------------------------------------------------------------------------+

$ openstack application credential list
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| ID                               | Name                  | Project ID                       | Description                 | Expires At |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack    | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles       | None       |
| 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None       |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+

3.2. Update the clouds.yaml with the openstack cloud and the new credential:
$ vim clouds.yaml
$ diff clouds.yaml clouds.yaml.bck
28c28
<  openstack:
---
>  shiftstack:
30,31c30,31
<         application_credential_id: 534373a90fde41f39da826a5d77a6441
<         application_credential_secret: _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA
---
>         application_credential_id: 339b768af8a44b1a95a5b30872a1a492
>         application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw

3.3. Upload the new clouds.yaml to the openstack-credentials secret in the kube-system namespace:
oc set data -n kube-system secret/openstack-credentials clouds.yaml="$(<~/clouds.yaml)"

3.4. Make sure the cred updated:
$ oc get secret -n kube-system openstack-credentials -o json | jq -r '.data."clouds.yaml"' | base64 -d


4. At this point operations with cinder csi are working fine.
Delete the project which includes cinder csi PVCs:
$ oc delete project topologyaware-test


5. Remove from openstack the old app cred:
$ openstack application credential list                                                                                                                                        
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| ID                               | Name                  | Project ID                       | Description                 | Expires At |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack    | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles       | None       |
| 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None       |                                                                        
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+                                                                        
(shiftstack) [stack@undercloud-0 ~]$ openstack application credential delete 339b768af8a44b1a95a5b30872a1a492


6. Apply a deployment with pvc: [1]
$ oc new-project topologyaware-test
$ oc apply -f cinder_csi_0.yaml
storageclass.storage.k8s.io/topology-aware-0 unchanged
persistentvolumeclaim/pvc-0 created
deployment.apps/demo-0 created


7. The kubelet tries to use the previous credentials and fails with an "Unauthorized" 401 code, and the pod stuck on ContainerCreating:
$ oc get pods
NAME                      READY   STATUS              RESTARTS   AGE
demo-0-6cd78b9fdc-2glz5   0/1     ContainerCreating   0          20m

$ oc describe pod demo-0-6cd78b9fdc-2glz5 | grep -A 20 Events
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               24m                  default-scheduler        Successfully assigned topologyaware-test/demo-0-6cd78b9fdc-2glz5 to ostest-pcw46-worker-0-s9zxj
  Normal   SuccessfulAttachVolume  24m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5"
  Warning  FailedMount             6m25s (x6 over 22m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[mydata kube-api-access-9t6h4]: timed out waiting for the condition
  Warning  FailedMount             111s (x4 over 17m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[kube-api-access-9t6h4 mydata]: timed out waiting for the condition
  Warning  FailedMount             100s (x19 over 24m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead
{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}

Unable to use application credentials for cinder-csi after OpenStack credentials update

Successful to use application credentials for cinder-csi after OpenStack credentials update

The 401 error is also present on the openshift-cloud-controller-manager:
$ oc logs -n openshift-cloud-controller-manager openstack-cloud-controller-manager-6f6f9c8979-tzfqz | grep 401 | tail -1
{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}