Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-5410

[AWS-EBS-CSI-Driver] provision volume using customer kms key couldn't restore its snapshot successfully

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 4.13
    • 4.13
    • Storage / Operators
    • None
    • Important
    • None
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * With this update, the default credentials request for AWS has been modified to allow customer-managed key to be used for re-encryption in the KMS. Administrators who created credentials requests in manual mode with CCO must apply those changes manually by adding "kms:ReEncrypt*" permission to their key policy. Other administrators are not impacted by this change. (link:https://issues.redhat.com/browse/OCPBUGS-5410[*OCPBUGS-5410*])
      Show
      * With this update, the default credentials request for AWS has been modified to allow customer-managed key to be used for re-encryption in the KMS. Administrators who created credentials requests in manual mode with CCO must apply those changes manually by adding "kms:ReEncrypt*" permission to their key policy. Other administrators are not impacted by this change. (link: https://issues.redhat.com/browse/OCPBUGS-5410 [* OCPBUGS-5410 *])
    • Bug Fix
    • Done

      Description of problem:

      [AWS-EBS-CSI-Driver] provision volume using customer kms key couldn't restore its snapshot successfully
      

      Version-Release number of selected component (if applicable):

      $ oc version
      Client Version: 4.12.0-ec.3
      Kustomize Version: v4.5.4
      Server Version: 4.13.0-0.nightly-2023-01-01-223309
      Kubernetes Version: v1.25.2+0003605
      
      I tested with 4.11.z and 4.12 nightly also have the same issue

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create aws ebs csi storageClass with customer managed kms key, volumeBindingMode: Immediate;
      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: my-kms-csi
      provisioner: ebs.csi.aws.com
      parameters:
        kmsKeyId: 'arn:aws:kms:us-east-2:301721915996:key/17e63c2f-0c10-4680-97a2-4664f974e2e4'
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      volumeBindingMode: Immediate
      
      2. Create pvc with the csi storageClass and after the volume provisioned succeed create snapshot for the volume with preset VolumeSnapshotClasse/csi-aws-vsc;
      # Origin pvc
      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: pvc-ori
        namespace: default
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: my-kms-csi
        volumeMode: Filesystem
      ---
      apiVersion: snapshot.storage.k8s.io/v1
      kind: VolumeSnapshot
      metadata:
        annotations:
          snapshot.storage.kubernetes.io/pvc-access-modes: ReadWriteOnce
          snapshot.storage.kubernetes.io/pvc-volume-mode: Filesystem
        name: pvc-ori-snapshot
      spec:
        source:
          persistentVolumeClaimName: pvc-ori
          volumeSnapshotClassName: csi-aws-vsc
      
      3. Waiting for the volumesnapshot/pvc-ori-snapshot ReadyToUse create pvc restore the snapshot with storageClass/my-kms-csi
      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: pvc-ori-restore
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: my-kms-csi
        volumeMode: Filesystem
        dataSource:
          apiGroup: snapshot.storage.k8s.io
          kind: VolumeSnapshot
          name: pvc-ori-snapshot
      
      4. Waiting for the restored volume provision succeed. 

      Actual results:

      In Step4 : The volume couldn't be provisioned successfully, pvc stuck at 'Pending'
      failed to provision volume with StorageClass "my-kms-csi": rpc error: code = Internal desc = Could not create volume "pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-002e6f75fc9d2e868' does not exist. status code: 400, request id: 2361646d-a9af-4bb2-a2e1-7268bf032292
      

      Expected results:

      In Step4 : The volume should be provisioned successfully

      Additional info:

      $ oc logs -l app=aws-ebs-csi-driver-controller -c csi-provisioner --tail=-1 | grep 'pvc-ori-restore'
      I0105 07:00:26.428554       1 controller.go:1337] provision "default/pvc-ori-restore" class "my-kms-csi": started
      I0105 07:00:26.428831       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/pvc-ori-restore"
      I0105 07:00:26.436091       1 connection.go:184] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}],"requisite":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","csi.storage.k8s.io/pvc/name":"pvc-ori-restore","csi.storage.k8s.io/pvc/namespace":"default","kmsKeyId":"arn:aws:kms:us-east-2:301721915996:key/17e63c2f-0c10-4680-97a2-4664f974e2e4"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"snap-0c3b1cb7358296c1f"}}}}
      I0105 07:00:29.892138       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "my-kms-csi": rpc error: code = Internal desc = Could not create volume "pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-002e6f75fc9d2e868' does not exist.
      I0105 07:00:30.893007       1 controller.go:1337] provision "default/pvc-ori-restore" class "my-kms-csi": started
      I0105 07:00:30.893113       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/pvc-ori-restore"
      I0105 07:00:30.899636       1 connection.go:184] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}],"requisite":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","csi.storage.k8s.io/pvc/name":"pvc-ori-restore","csi.storage.k8s.io/pvc/namespace":"default","kmsKeyId":"arn:aws:kms:us-east-2:301721915996:key/17e63c2f-0c10-4680-97a2-4664f974e2e4"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"snap-0c3b1cb7358296c1f"}}}}
      I0105 07:00:30.902068       1 round_trippers.go:553] PATCH https://172.30.0.1:443/api/v1/namespaces/default/events/pvc-ori-restore.17375787954e8b58 200 OK in 8 milliseconds
      I0105 07:00:31.207107       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "my-kms-csi": rpc error: code = AlreadyExists desc = Could not create volume "pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)

            rbednar@redhat.com Roman Bednar
            rhn-support-pewang Penghao Wang
            Penghao Wang Penghao Wang
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: