-
Bug
-
Resolution: Done
-
Major
-
None
-
4.12.0
-
?
-
Important
-
None
-
ShiftStack Sprint 230
-
1
-
Rejected
-
False
-
-
-
Rejected
This is a clone of issue OCPBUGS-4103. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-3774. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-2845. The following is the description of the original issue:
—
Description of problem:
After OpenStack application credentials update, Cinder CSI tries to use the previous credentials and fails with an "Unauthorized" 401 code:
MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}
Version-Release number of selected component (if applicable):
4.12.0-0.nightly-2022-10-25-07574 on top of RHOS-16.2-RHEL-8-20220804.n.1
How reproducible:
Always and with all of the network types
Steps to Reproduce:
1. Install OCP 4.12 with application credentials on top of OSP:
$ source shiftstackrc && openstack application credential list
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| ID | Name | Project ID | Description | Expires At |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles | None |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
$ grep -A 10 shiftstack clouds.yaml
#BEGIN shiftstack PARAMETERS
shiftstack:
auth:
application_credential_id: 339b768af8a44b1a95a5b30872a1a492
application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw
auth_url: https://10.46.44.140:13000
auth_type: v3applicationcredential
cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem
identity_api_version: '3'
region_name: regionOne
#END shiftstack PARAMETERS
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.12.0-0.nightly-2022-10-25-075747 True False 26m Cluster version is 4.12.0-0.nightly-2022-10-25-075747
2. Deploy a namespace with PVC and a cinder.csi.openstack.org StorageClass:
$ oc project topologyaware-test
Now using project "topologyaware-test" on server "https://api.ostest.shiftstack.com:6443".
(shiftstack) [stack@undercloud-0 ~]$ oc get pods
NAME READY STATUS RESTARTS AGE
demo-0-6cd78b9fdc-w6xf6 1/1 Running 0 4m15s
demo-1-fdc6c877c-5pcpp 1/1 Running 0 4m8s
demo-2-f8c498bbc-pdl4r 1/1 Running 0 4m2s
$ oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-manila-default manila.csi.openstack.org Delete Immediate false 59m
standard-csi (default) cinder.csi.openstack.org Delete WaitForFirstConsumer true 59m
topology-aware-0 cinder.csi.openstack.org Delete WaitForFirstConsumer false 5m33s
topology-aware-1 cinder.csi.openstack.org Delete WaitForFirstConsumer false 5m16s
topology-aware-2 cinder.csi.openstack.org Delete WaitForFirstConsumer false 4m55s
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-0 Bound pvc-12eede08-ebb0-4ced-8d1e-3aa8dc6ec7c9 1Gi RWO topology-aware-0 5m40s
pvc-1 Bound pvc-0ed7386f-8962-4975-ad28-21ac8dcacb0b 1Gi RWO topology-aware-1 5m23s
pvc-2 Bound pvc-5f50bb91-1b8e-467f-8a06-577d7750bda2 1Gi RWO topology-aware-2 5m2s
3. Update OpenStack application credentials [0]
3.1. Create a new application credential:
$ openstack application credential create --description "App Creds - All roles - New" AppCredsShiftstackNew
+--------------+----------------------------------------------------------------------------------------+
| Field | Value |
+--------------+----------------------------------------------------------------------------------------+
| description | App Creds - All roles - New |
| expires_at | None |
| id | 534373a90fde41f39da826a5d77a6441 |
| name | AppCredsShiftstackNew |
| project_id | 2073ea9b31cd4a17a72046da0f6e002c |
| roles | swiftoperator member reader |
| secret | _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA |
| system | None |
| unrestricted | False |
| user_id | 67ac336163bb4606ac3ade19454c9089 |
+--------------+----------------------------------------------------------------------------------------+
$ openstack application credential list
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| ID | Name | Project ID | Description | Expires At |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles | None |
| 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
3.2. Update the clouds.yaml with the openstack cloud and the new credential:
$ vim clouds.yaml
$ diff clouds.yaml clouds.yaml.bck
28c28
< openstack:
---
> shiftstack:
30,31c30,31
< application_credential_id: 534373a90fde41f39da826a5d77a6441
< application_credential_secret: _gdWJp25vD1jb0f6FpM2Ax-nYbOQd5qENNQmE67D7yU6aWLOTIXTKF9g12AaQJu8Js2NMEQosJ6BZZpYLlTcLA
---
> application_credential_id: 339b768af8a44b1a95a5b30872a1a492
> application_credential_secret: MbH8ZUEFVgkLWDMqzUVgrfPqEsg4KRFqfifrJF5TwhlhFyP-mp3xK0L-VoQuPOkAmRu1MmIjoYORBsu9qlbOLw
3.3. Upload the new clouds.yaml to the openstack-credentials secret in the kube-system namespace:
oc set data -n kube-system secret/openstack-credentials clouds.yaml="$(<~/clouds.yaml)"
3.4. Make sure the cred updated:
$ oc get secret -n kube-system openstack-credentials -o json | jq -r '.data."clouds.yaml"' | base64 -d
4. At this point operations with cinder csi are working fine.
Delete the project which includes cinder csi PVCs:
$ oc delete project topologyaware-test
5. Remove from openstack the old app cred:
$ openstack application credential list
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| ID | Name | Project ID | Description | Expires At |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
| 339b768af8a44b1a95a5b30872a1a492 | AppCredsshiftstack | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles | None |
| 534373a90fde41f39da826a5d77a6441 | AppCredsShiftstackNew | 2073ea9b31cd4a17a72046da0f6e002c | App Creds - All roles - New | None |
+----------------------------------+-----------------------+----------------------------------+-----------------------------+------------+
(shiftstack) [stack@undercloud-0 ~]$ openstack application credential delete 339b768af8a44b1a95a5b30872a1a492
6. Apply a deployment with pvc: [1]
$ oc new-project topologyaware-test
$ oc apply -f cinder_csi_0.yaml
storageclass.storage.k8s.io/topology-aware-0 unchanged
persistentvolumeclaim/pvc-0 created
deployment.apps/demo-0 created
7. The kubelet tries to use the previous credentials and fails with an "Unauthorized" 401 code, and the pod stuck on ContainerCreating:
$ oc get pods
NAME READY STATUS RESTARTS AGE
demo-0-6cd78b9fdc-2glz5 0/1 ContainerCreating 0 20m
$ oc describe pod demo-0-6cd78b9fdc-2glz5 | grep -A 20 Events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 24m default-scheduler Successfully assigned topologyaware-test/demo-0-6cd78b9fdc-2glz5 to ostest-pcw46-worker-0-s9zxj
Normal SuccessfulAttachVolume 24m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5"
Warning FailedMount 6m25s (x6 over 22m) kubelet Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[mydata kube-api-access-9t6h4]: timed out waiting for the condition
Warning FailedMount 111s (x4 over 17m) kubelet Unable to attach or mount volumes: unmounted volumes=[mydata], unattached volumes=[kube-api-access-9t6h4 mydata]: timed out waiting for the condition
Warning FailedMount 100s (x19 over 24m) kubelet MountVolume.MountDevice failed for volume "pvc-b5987bc9-54ee-444d-a65b-fd0b584b6be5" : rpc error: code = Internal desc = GetVolume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://10.46.44.140:13776/v3/2073ea9b31cd4a17a72046da0f6e002c/volumes/df6e1aba-2528-4405-a0d8-9b71fedbd49c], but got 401 instead
{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}
Actual results:
Unable to use application credentials for cinder-csi after OpenStack credentials update
Expected results:
Successful to use application credentials for cinder-csi after OpenStack credentials update
Additional info:
The 401 error is also present on the openshift-cloud-controller-manager:
$ oc logs -n openshift-cloud-controller-manager openstack-cloud-controller-manager-6f6f9c8979-tzfqz | grep 401 | tail -1
{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://10.46.44.140:13000/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: 339b768af8a44b1a95a5b30872a1a492.","title":"Not Found"}}
Attached [^must-gather.tar.gz]
- clones
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- links to