Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-35183

[4.14] The secrets-store-csi-driver with AWS provider integration does not work in HyperShift hosted cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • 4.14.z
    • 4.14
    • Storage / Operators
    • None
    • Critical
    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the secrets-store CSI driver on Hypershift was failing to mount secrets due to an issue with the Hypershift CLI. With this release, the driver is able to mount volumes and the issue has been resolved. (link:https://issues.redhat.com/browse/OCPBUGS-35183[*OCPBUGS-35183*])
      _________
      Secrets store CSI driver on Hypershift was failing to mount secrets due to a bug in hypershift cli that creates OIDC infrastructure. This has been fixed and the driver is now able to mount volumes.
      Show
      * Previously, the secrets-store CSI driver on Hypershift was failing to mount secrets due to an issue with the Hypershift CLI. With this release, the driver is able to mount volumes and the issue has been resolved. (link: https://issues.redhat.com/browse/OCPBUGS-35183 [* OCPBUGS-35183 *]) _________ Secrets store CSI driver on Hypershift was failing to mount secrets due to a bug in hypershift cli that creates OIDC infrastructure. This has been fixed and the driver is now able to mount volumes.
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-34997. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-34759. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-18711. The following is the description of the original issue:

      Description of problem:

      secrets-store-csi-driver with AWS provider does not work in HyperShift hosted cluster, pod can't mount the volume successfully.

      Version-Release number of selected component (if applicable):

      secrets-store-csi-driver-operator.v4.14.0-202308281544 in 4.14.0-0.nightly-2023-09-06-235710 HyperShift hosted cluster.

      How reproducible:

      Always

      Steps to Reproduce:

      1. Follow test case OCP-66032 "Setup" part to install secrets-store-csi-driver-operator.v4.14.0-202308281544 , secrets-store-csi-driver and AWS provider successfully:
      
      $ oc get po -n openshift-cluster-csi-drivers
      NAME                                                READY   STATUS    RESTARTS   AGE
      aws-ebs-csi-driver-node-7xxgr                       3/3     Running   0          5h18m
      aws-ebs-csi-driver-node-fmzwf                       3/3     Running   0          5h18m
      aws-ebs-csi-driver-node-rgrxd                       3/3     Running   0          5h18m
      aws-ebs-csi-driver-node-tpcxq                       3/3     Running   0          5h18m
      csi-secrets-store-provider-aws-2fm6q                1/1     Running   0          5m14s
      csi-secrets-store-provider-aws-9xtw7                1/1     Running   0          5m15s
      csi-secrets-store-provider-aws-q5lvb                1/1     Running   0          5m15s
      csi-secrets-store-provider-aws-q6m65                1/1     Running   0          5m15s
      secrets-store-csi-driver-node-4wdc8                 3/3     Running   0          6m22s
      secrets-store-csi-driver-node-n7gkj                 3/3     Running   0          6m23s
      secrets-store-csi-driver-node-xqr52                 3/3     Running   0          6m22s
      secrets-store-csi-driver-node-xr24v                 3/3     Running   0          6m22s
      secrets-store-csi-driver-operator-9cb55b76f-7cbvz   1/1     Running   0          7m16s
      
      2. Follow test case OCP-66032 steps to create AWS secret, set up AWS IRSA successfully.
      
      3. Follow test case OCP-66032 steps SecretProviderClass, deployment with the secretProviderClass successfully. Then check pod, pod is stuck in ContainerCreating:
      
      $ oc get po
      NAME                               READY   STATUS              RESTARTS   AGE
      hello-openshift-84c76c5b89-p5k4f   0/1     ContainerCreating   0          10m
      
      $ oc describe po hello-openshift-84c76c5b89-p5k4f
      ...
      Events:
        Type     Reason       Age   From               Message
        ----     ------       ----  ----               -------
        Normal   Scheduled    11m   default-scheduler  Successfully assigned xxia-proj/hello-openshift-84c76c5b89-p5k4f to ip-10-0-136-205.us-east-2.compute.internal
        Warning  FailedMount  11m   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 92d1ff5b-36be-4cc5-9b55-b12279edd78e
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 50907328-70a6-44e0-9f05-80a31acef0b4
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 617dc3bc-a5e3-47b0-b37c-825f8dd84920
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 8ab5fc2c-00ca-45e2-9a82-7b1765a5df1a
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: b76019ca-dc04-4e3e-a305-6db902b0a863
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: b395e3b2-52a2-4fc2-80c6-9a9722e26375
        Warning  FailedMount  11m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: ec325057-9c0a-4327-80c9-a9b6233a64dd
        Warning  FailedMount  10m  kubelet  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
                 status code: 400, request id: 405492b2-ed52-429b-b253-6a7c098c26cb
        Warning  FailedMount  82s (x5 over 9m35s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
        Warning  FailedMount  74s (x5 over 9m25s)  kubelet  (combined from similar events): MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxia-proj/hello-openshift-84c76c5b89-p5k4f, err: rpc error: code = Unknown desc = us-east-2: Failed fetching secret xxiaSecret: WebIdentityErr: failed to retrieve credentials
      caused by: InvalidIdentityToken: Incorrect token audience
        status code: 400, request id: c38bbed1-012d-4250-b674-24ab40607920

      Actual results:

      Hit above stuck issue.

      Expected results:

      Pod should be Running.

      Additional info:

      Compared another operator (cert-manager-operator) which also uses AWS IRSA: OCP-62500 , that case works well. So secrets-store-csi-driver-operator has bug.

              rbednar@redhat.com Roman Bednar
              openshift-crt-jira-prow OpenShift Prow Bot
              Rohit Patil Rohit Patil
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: