After OpenStack application credentials update, Cinder CSI tries to use the previous credentials and fails with an "Unauthorized" 401 code:

MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}

4.12.0-0.nightly-2022-10-25-07574 on top of RHOS-16.2-RHEL-8-20220804.n.1

Always and with all of the network types

1. Install OCP 4.12 with application credentials on top of OSP:
$ source shiftstackrc && openstack application credential list
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| ID                               | Name               | Project ID                       | Description           | Expires At |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles | None       |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+

$ grep -A 10 shiftstack clouds.yaml
#BEGIN shiftstack PARAMETERS
 shiftstack:
    auth:
        application_credential_id: 339b768af8a44b1a95a5b30872a1a492
        application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw
        auth_url: https://10.46.44.140:13000
    auth_type: v3applicationcredential
    cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem
    identity_api_version: '3'
    region_name: regionOne
#END shiftstack PARAMETERS

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-10-25-075747   True        False         26m     Cluster version is 4.12.0-0.nightly-2022-10-25-075747


2. Deploy a namespace with PVC and a cinder.csi.openstack.org StorageClass:
$ oc project topologyaware-test
Now using project "topologyaware-test" on server "https://api.ostest.shiftstack.com:6443".
(shiftstack) [stack@undercloud-0 ~]$ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
demo-0-6cd78b9fdc-w6xf6   1/1     Running   0          4m15s
demo-1-fdc6c877c-5pcpp    1/1     Running   0          4m8s
demo-2-f8c498bbc-pdl4r    1/1     Running   0          4m2s
$ oc get sc
NAME                     PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-default       manila.csi.openstack.org   Delete          Immediate              false                  59m
standard-csi (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   59m
topology-aware-0         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  5m33s
topology-aware-1         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  5m16s
topology-aware-2         cinder.csi.openstack.org   Delete          WaitForFirstConsumer   false                  4m55s
$ oc get pvc
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
pvc-0   Bound    pvc-12eede08-ebb0-4ced-8d1e-3aa8dc6ec7c9   1Gi        RWO            topology-aware-0   5m40s
pvc-1   Bound    pvc-0ed7386f-8962-4975-ad28-21ac8dcacb0b   1Gi        RWO            topology-aware-1   5m23s
pvc-2   Bound    pvc-5f50bb91-1b8e-467f-8a06-577d7750bda2   1Gi        RWO            topology-aware-2   5m2s


3. Update OpenStack application credentials [0]
3.1. Create a new application credential:
$ openstack application credential create --description "App Creds - All roles - New" AppCredsShiftstackNew
+--------------+----------------------------------------------------------------------------------------+
| Field        | Value                                                                                  |
+--------------+----------------------------------------------------------------------------------------+
| description  | App Creds - All roles - New                                                            |
| expires_at   | None                                                                                   |
| id           | 534373a90fde41f39da826a5d77a6441                                                       |
| name         | AppCredsShiftstackNew                                                                  |
| project_id   | 2073ea9b31cd4a17a72046da0f6e002c                                                       |
| roles        | swiftoperator member reader                                                            |
| secret       | _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA |
| system       | None                                                                                   |
| unrestricted | False                                                                                  |
| user_id      | 67ac336163bb4606ac3ade19454c9089                                                       |
+--------------+----------------------------------------------------------------------------------------+

$ openstack application credential list
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| ID                               | Name                  | Project ID                       | Description                 | Expires At |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack    | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles       | None       |
| 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None       |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+

3.2. Update the clouds.yaml with the openstack cloud and the new credential:
$ vim clouds.yaml
$ diff clouds.yaml clouds.yaml.bck
28c28
<  openstack:
---
>  shiftstack:
30,31c30,31
<         application_credential_id: 534373a90fde41f39da826a5d77a6441
<         application_credential_secret: _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA
---
>         application_credential_id: 339b768af8a44b1a95a5b30872a1a492
>         application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw

3.3. Upload the new clouds.yaml to the openstack-credentials secret in the kube-system namespace:
oc set data -n kube-system secret/openstack-credentials clouds.yaml="$(<~/clouds.yaml)"

3.4. Make sure the cred updated:
$ oc get secret -n kube-system openstack-credentials -o json | jq -r '.data."clouds.yaml"' | base64 -d


4. At this point operations with cinder csi are working fine.
Delete the project which includes cinder csi PVCs:
$ oc delete project topologyaware-test


5. Remove from openstack the old app cred:
$ openstack application credential list                                                                                                                                        
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| ID                               | Name                  | Project ID                       | Description                 | Expires At |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack    | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles       | None       |
| 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None       |                                                                        
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+                                                                        
(shiftstack) [stack@undercloud-0 ~]$ openstack application credential delete 339b768af8a44b1a95a5b30872a1a492


6. Apply a deployment with pvc: [1]
$ oc new-project topologyaware-test
$ oc apply -f cinder_csi_0.yaml
storageclass.storage.k8s.io/topology-aware-0 unchanged
persistentvolumeclaim/pvc-0 created
deployment.apps/demo-0 created


7. The kubelet tries to use the previous credentials and fails with an "Unauthorized" 401 code, and the pod stuck on ContainerCreating:
$ oc get pods
NAME                      READY   STATUS              RESTARTS   AGE
demo-0-6cd78b9fdc-2glz5   0/1     ContainerCreating   0          20m

$ oc describe pod demo-0-6cd78b9fdc-2glz5 | grep -A 20 Events
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               24m                  default-scheduler        Successfully assigned topologyaware-test/demo-0-6cd78b9fdc-2glz5 to ostest-pcw46-worker-0-s9zxj
  Normal   SuccessfulAttachVolume  24m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5"
  Warning  FailedMount             6m25s (x6 over 22m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[mydata kube-api-access-9t6h4]: timed out waiting for the condition
  Warning  FailedMount             111s (x4 over 17m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[kube-api-access-9t6h4 mydata]: timed out waiting for the condition
  Warning  FailedMount             100s (x19 over 24m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead
{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}

Unable to use application credentials for cinder-csi after OpenStack credentials update

Successful to use application credentials for cinder-csi after OpenStack credentials update

The 401 error is also present on the openshift-cloud-controller-manager:
$ oc logs -n openshift-cloud-controller-manager openstack-cloud-controller-manager-6f6f9c8979-tzfqz | grep 401 | tail -1
{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}