-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
4.14
-
None
-
Critical
-
No
-
False
-
-
Secrets store CSI driver on Hypershift was failing to mount secrets due to a bug in hypershift cli that creates OIDC infrastructure. This has been fixed and the driver is now able to mount volumes.
-
Bug Fix
-
In Progress
This is a clone of issue OCPBUGS-18711. The following is the description of the original issue:
—
Description of problem:
secrets-store-csi-driver with AWS provider does not work in HyperShift hosted cluster, pod can't mount the volume successfully.
Version-Release number of selected component (if applicable):
secrets-store-csi-driver-operator.v4.14.0-202308281544 in 4.14.0-0.nightly-2023-09-06-235710 HyperShift hosted cluster.
How reproducible:
Always
Steps to Reproduce:
1. Follow test case OCP-66032 "Setup" part to install secrets-store-csi-driver-operator.v4.14.0-202308281544 , secrets-store-csi-driver and AWS provider successfully: $ oc get po -n openshift-cluster-csi-drivers NAME READY STATUS RESTARTS AGE aws-ebs-csi-driver-node-7xxgr 3/3 Running 0 5h18m aws-ebs-csi-driver-node-fmzwf 3/3 Running 0 5h18m aws-ebs-csi-driver-node-rgrxd 3/3 Running 0 5h18m aws-ebs-csi-driver-node-tpcxq 3/3 Running 0 5h18m csi-secrets-store-provider-aws-2fm6q 1/1 Running 0 5m14s csi-secrets-store-provider-aws-9xtw7 1/1 Running 0 5m15s csi-secrets-store-provider-aws-q5lvb 1/1 Running 0 5m15s csi-secrets-store-provider-aws-q6m65 1/1 Running 0 5m15s secrets-store-csi-driver-node-4wdc8 3/3 Running 0 6m22s secrets-store-csi-driver-node-n7gkj 3/3 Running 0 6m23s secrets-store-csi-driver-node-xqr52 3/3 Running 0 6m22s secrets-store-csi-driver-node-xr24v 3/3 Running 0 6m22s secrets-store-csi-driver-operator-9cb55b76f-7cbvz 1/1 Running 0 7m16s 2. Follow test case OCP-66032 steps to create AWS secret, set up AWS IRSA successfully. 3. Follow test case OCP-66032 steps SecretProviderClass, deployment with the secretProviderClass successfully. Then check pod, pod is stuck in ContainerCreating: $ oc get po NAME READY STATUS RESTARTS AGE hello-openshift-84c76c5b89-p5k4f 0/1 ContainerCreating 0 10m $ oc describe po hello-openshift-84c76c5b89-p5k4f ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 11m default-scheduler Successfully assigned xxia-proj/hello-openshift-84c76c5b89-p5k4f to ip-10-0-136-205.us-east-2.compute.internal Warning FailedMount 11m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: 92d1ff5b-36be-4cc5-9b55-b12279edd78e Warning FailedMount 11m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: 50907328-70a6-44e0-9f05-80a31acef0b4 Warning FailedMount 11m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: 617dc3bc-a5e3-47b0-b37c-825f8dd84920 Warning FailedMount 11m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: 8ab5fc2c-00ca-45e2-9a82-7b1765a5df1a Warning FailedMount 11m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: b76019ca-dc04-4e3e-a305-6db902b0a863 Warning FailedMount 11m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: b395e3b2-52a2-4fc2-80c6-9a9722e26375 Warning FailedMount 11m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: ec325057-9c0a-4327-80c9-a9b6233a64dd Warning FailedMount 10m kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: 405492b2-ed52-429b-b253-6a7c098c26cb Warning FailedMount 82s (x5 over 9m35s) kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition Warning FailedMount 74s (x5 over 9m25s) kubelet (combined from similar events): MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Incorrect token audience status code: 400, request id: c38bbed1-012d-4250-b674-24ab40607920
Actual results:
Hit above stuck issue.
Expected results:
Pod should be Running.
Additional info:
Compared another operator (cert-manager-operator) which also uses AWS IRSA: OCP-62500 , that case works well. So secrets-store-csi-driver-operator has bug.
- clones
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- links to
