[AWS-EBS-CSI-Driver] provision volume using customer kms key couldn't restore its snapshot successfully

[AWS-EBS-CSI-Driver] provision volume using customer kms key couldn't restore its snapshot successfully

$ oc version
Client Version: 4.12.0-ec.3
Kustomize Version: v4.5.4
Server Version: 4.13.0-0.nightly-2023-01-01-223309
Kubernetes Version: v1.25.2+0003605

I tested with 4.11.z and 4.12 nightly also have the same issue

Always

1. Create aws ebs csi storageClass with customer managed kms key, volumeBindingMode: Immediate;
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: my-kms-csi
provisioner: ebs.csi.aws.com
parameters:
  kmsKeyId: 'arn:aws:kms:us-east-2:301721915996:key/17e63c2f-0c10-4680-97a2-4664f974e2e4'
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

2. Create pvc with the csi storageClass and after the volume provisioned succeed create snapshot for the volume with preset VolumeSnapshotClasse/csi-aws-vsc;
# Origin pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-ori
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: my-kms-csi
  volumeMode: Filesystem
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  annotations:
    snapshot.storage.kubernetes.io/pvc-access-modes: ReadWriteOnce
    snapshot.storage.kubernetes.io/pvc-volume-mode: Filesystem
  name: pvc-ori-snapshot
spec:
  source:
    persistentVolumeClaimName: pvc-ori
    volumeSnapshotClassName: csi-aws-vsc

3. Waiting for the volumesnapshot/pvc-ori-snapshot ReadyToUse create pvc restore the snapshot with storageClass/my-kms-csi
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-ori-restore
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: my-kms-csi
  volumeMode: Filesystem
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: pvc-ori-snapshot

4. Waiting for the restored volume provision succeed. 

In Step4 : The volume couldn't be provisioned successfully, pvc stuck at 'Pending'
failed to provision volume with StorageClass "my-kms-csi": rpc error: code = Internal desc = Could not create volume "pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-002e6f75fc9d2e868' does not exist. status code: 400, request id: 2361646d-a9af-4bb2-a2e1-7268bf032292

In Step4 : The volume should be provisioned successfully

$ oc logs -l app=aws-ebs-csi-driver-controller -c csi-provisioner --tail=-1 | grep 'pvc-ori-restore'
I0105 07:00:26.428554       1 controller.go:1337] provision "default/pvc-ori-restore" class "my-kms-csi": started
I0105 07:00:26.428831       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/pvc-ori-restore"
I0105 07:00:26.436091       1 connection.go:184] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}],"requisite":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","csi.storage.k8s.io/pvc/name":"pvc-ori-restore","csi.storage.k8s.io/pvc/namespace":"default","kmsKeyId":"arn:aws:kms:us-east-2:301721915996:key/17e63c2f-0c10-4680-97a2-4664f974e2e4"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"snap-0c3b1cb7358296c1f"}}}}
I0105 07:00:29.892138       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "my-kms-csi": rpc error: code = Internal desc = Could not create volume "pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-002e6f75fc9d2e868' does not exist.
I0105 07:00:30.893007       1 controller.go:1337] provision "default/pvc-ori-restore" class "my-kms-csi": started
I0105 07:00:30.893113       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/pvc-ori-restore"
I0105 07:00:30.899636       1 connection.go:184] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}],"requisite":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2c"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a","csi.storage.k8s.io/pvc/name":"pvc-ori-restore","csi.storage.k8s.io/pvc/namespace":"default","kmsKeyId":"arn:aws:kms:us-east-2:301721915996:key/17e63c2f-0c10-4680-97a2-4664f974e2e4"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"snap-0c3b1cb7358296c1f"}}}}
I0105 07:00:30.902068       1 round_trippers.go:553] PATCH https://172.30.0.1:443/api/v1/namespaces/default/events/pvc-ori-restore.17375787954e8b58 200 OK in 8 milliseconds
I0105 07:00:31.207107       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-ori-restore", UID:"a1dd6aa6-1339-4cf1-9e10-16580e00ef0a", APIVersion:"v1", ResourceVersion:"170970", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "my-kms-csi": rpc error: code = AlreadyExists desc = Could not create volume "pvc-a1dd6aa6-1339-4cf1-9e10-16580e00ef0a": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)