HCP deletion gets stuck when additional security groups are included in the nodepool.
  The latest 4.16 CLI + 4.16 HO + 4.16.0-0.nightly-2024-02-22-021321 OCP are used for testing.

    platform:
    aws:
      instanceProfile: jie-cluster-worker
      instanceType: m5.large
      rootVolume:
        size: 120
        type: gp3
      securityGroups:
        - id: sg-0f9a5a27c90c1dce4
        - id: sg-011c554498c068367
      subnet:
        id: subnet-08ff26edbcadd8561
    type: AWS

jiezhao-mac:hypershift jiezhao$ ./bin/hypershift destroy cluster aws --name jie-test --aws-creds $HOME/.aws/credentials --destroy-cloud-resources --base-domain=jiezhao-dev.qe.devcluster.openshift.com --infra-id=jie-cluster
2024-02-22T12:04:51-05:00    INFO    Found hosted cluster    {"namespace": "clusters", "name": "jie-test"}
2024-02-22T12:04:51-05:00    INFO    Setting client finalizer for hosted cluster    {"namespace": "clusters", "name": "jie-test"}
2024-02-22T12:04:51-05:00    INFO    Marking cleanup of cloud resources for hosted cluster    {"namespace": "clusters", "name": "jie-test"}
2024-02-22T12:04:51-05:00    INFO    Updated hosted cluster    {"namespace": "clusters", "name": "jie-test"}
2024-02-22T12:04:51-05:00    INFO    Deleting hosted cluster    {"namespace": "clusters", "name": "jie-test"}
2024-02-22T12:14:51-05:00    ERROR    Failed to get hosted cluster    {"namespace": "clusters", "name": "jie-test", "error": "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline"}
github.com/openshift/hypershift/cmd/cluster/core.waitForRestOfFinalizers.func1
    /Users/jiezhao/hypershift-test/hypershift/cmd/cluster/core/destroy.go:224
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
    /Users/jiezhao/hypershift-test/hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
    /Users/jiezhao/hypershift-test/hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
    /Users/jiezhao/hypershift-test/hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:33
github.com/openshift/hypershift/cmd/cluster/core.waitForRestOfFinalizers
    /Users/jiezhao/hypershift-test/hypershift/cmd/cluster/core/destroy.go:219
github.com/openshift/hypershift/cmd/cluster/core.DestroyCluster
    /Users/jiezhao/hypershift-test/hypershift/cmd/cluster/core/destroy.go:140
github.com/openshift/hypershift/cmd/cluster/aws.DestroyCluster
    /Users/jiezhao/hypershift-test/hypershift/cmd/cluster/aws/destroy.go:138
github.com/openshift/hypershift/cmd/cluster/aws.NewDestroyCommand.func1
    /Users/jiezhao/hypershift-test/hypershift/cmd/cluster/aws/destroy.go:42
github.com/spf13/cobra.(*Command).execute
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:1115
github.com/spf13/cobra.(*Command).Execute
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:1032
main.main
    /Users/jiezhao/hypershift-test/hypershift/main.go:78
runtime.main
    /usr/local/Cellar/go/1.20.4/libexec/src/runtime/proc.go:250
2024-02-22T12:14:51-05:00    ERROR    Failed to destroy cluster    {"error": "hostedcluster wasn't finalized, aborting delete: context deadline exceeded"}
github.com/openshift/hypershift/cmd/cluster/aws.NewDestroyCommand.func1
    /Users/jiezhao/hypershift-test/hypershift/cmd/cluster/aws/destroy.go:43
github.com/spf13/cobra.(*Command).execute
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:1115
github.com/spf13/cobra.(*Command).Execute
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext
    /Users/jiezhao/hypershift-test/hypershift/vendor/github.com/spf13/cobra/command.go:1032
main.main
    /Users/jiezhao/hypershift-test/hypershift/main.go:78
runtime.main
    /usr/local/Cellar/go/1.20.4/libexec/src/runtime/proc.go:250
Error: hostedcluster wasn't finalized, aborting delete: context deadline exceeded
hostedcluster wasn't finalized, aborting delete: context deadline exceeded
jiezhao-mac:hypershift jiezhao$ 

    always

    1. Create a cluster
    2. create a nodepool with two additional security groups
    3. Destroy the cluster using destroy command
    
    

    Destroy fails

    Destroy succeeds

 After destroy fails, manually delete two additional security groups on AWS console, destroy the cluster again using destroy command, destroy succeeds