Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18711

The secrets-store-csi-driver with AWS provider integration does not work in HyperShift hosted cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.14
    • Storage / Operators
    • None
    • Critical
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Secrets store CSI driver on Hypershift was failing to mount secrets due to a bug in hypershift cli that creates OIDC infrastructure. This has been fixed and the driver is now able to mount volumes.
    • Bug Fix
    • In Progress

      Description of problem:

      secrets-store-csi-driver with AWS provider does not work in HyperShift hosted cluster, pod can't mount the volume successfully.

      Version-Release number of selected component (if applicable):

      secrets-store-csi-driver-operator.v4.14.0-202308281544 in 4.14.0-0.nightly-2023-09-06-235710 HyperShift hosted cluster.

      How reproducible:

      Always

      Steps to Reproduce:

      1. Follow test case OCP-66032 "Setup" part to install secrets-store-csi-driver-operator.v4.14.0-202308281544 , secrets-store-csi-driver and AWS provider successfully:
      
      $ oc get po -n openshift-cluster-csi-drivers
      NAME                                                READY   STATUS    RESTARTS   AGE
      aws-ebs-csi-driver-node-7xxgr                       3/3     Running   0          5h18m
      aws-ebs-csi-driver-node-fmzwf                       3/3     Running   0          5h18m
      aws-ebs-csi-driver-node-rgrxd                       3/3     Running   0          5h18m
      aws-ebs-csi-driver-node-tpcxq                       3/3     Running   0          5h18m
      csi-secrets-store-provider-aws-2fm6q                1/1     Running   0          5m14s
      csi-secrets-store-provider-aws-9xtw7                1/1     Running   0          5m15s
      csi-secrets-store-provider-aws-q5lvb                1/1     Running   0          5m15s
      csi-secrets-store-provider-aws-q6m65                1/1     Running   0          5m15s
      secrets-store-csi-driver-node-4wdc8                 3/3     Running   0          6m22s
      secrets-store-csi-driver-node-n7gkj                 3/3     Running   0          6m23s
      secrets-store-csi-driver-node-xqr52                 3/3     Running   0          6m22s
      secrets-store-csi-driver-node-xr24v                 3/3     Running   0          6m22s
      secrets-store-csi-driver-operator-9cb55b76f-7cbvz   1/1     Running   0          7m16s
      
      2. Follow test case OCP-66032 steps to create AWS secret, set up AWS IRSA successfully.
      
      3. Follow test case OCP-66032 steps SecretProviderClass, deployment with the secretProviderClass successfully. Then check pod, pod is stuck in ContainerCreating:
      
      $ oc get po
      NAME                               READY   STATUS              RESTARTS   AGE
      hello-openshift-84c76c5b89-p5k4f   0/1     ContainerCreating   0          10m
      
      $ oc describe po hello-openshift-84c76c5b89-p5k4f
      ...
      Events:
        Type     Reason       Age   From               Message
        ----     ------       ----  ----               -------
        Normal   Scheduled    11m   default-scheduler  Successfully assigned xxia-proj/hello-openshift-84c76c5b89-p5k4f to ip-10-0-136-205.us-east-2.compute.internal
        Warning  FailedMount  11m   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 92d1ff5b-36be-4cc5-9b55-b12279edd78e
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 50907328-70a6-44e0-9f05-80a31acef0b4
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 617dc3bc-a5e3-47b0-b37c-825f8dd84920
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 8ab5fc2c-00ca-45e2-9a82-7b1765a5df1a
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: b76019ca-dc04-4e3e-a305-6db902b0a863
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: b395e3b2-52a2-4fc2-80c6-9a9722e26375
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: ec325057-9c0a-4327-80c9-a9b6233a64dd
        Warning  FailedMount  10m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 405492b2-ed52-429b-b253-6a7c098c26cb
        Warning  FailedMount  82s (x5 over 9m35s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
        Warning  FailedMount  74s (x5 over 9m25s)  kubelet  (combined from similar events): MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
        status code: 400, request id: c38bbed1-012d-4250-b674-24ab40607920

      Actual results:

      Hit above stuck issue.

      Expected results:

      Pod should be Running.

      Additional info:

      Compared another operator (cert-manager-operator) which also uses AWS IRSA: OCP-62500 , that case works well. So secrets-store-csi-driver-operator has bug.

            rbednar@redhat.com Roman Bednar
            xxia-1 Xingxing Xia
            Rohit Patil Rohit Patil
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: