Deleting hostedcluster with the "hcp destroy cluster agent" command is often failing, especially when using the boot-it-yourself flow

OCP 4.15.0-rc.8
MCE 2.5.0-DOWNANDBACK-2024-02-20-00-10-33

50% of the times when we use boot-it-yourself flow. Less often when using BMHs

    1. Install hosted cluster using boot-it-yourself flow to add workers
    2. Destroy the hosted cluster by using the CLI by following the documentation at https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.9/html/clusters/cluster_mce_overview#hypershift-cluster-destroy-bm-cli
    3. Delete the managedcluster resource
    4. Delete the hostedcluster using the command: hcp destroy cluster agent --name <cluster_name> 

$ ./hypershift_working/hypershift/bin/hcp destroy cluster agent --name hosted-0 --cluster-grace-period 20m0s
2024-02-22T09:36:19-05:00    INFO    Found hosted cluster    {"namespace": "clusters", "name": "hosted-0"}
2024-02-22T09:36:19-05:00    INFO    Deleting hosted cluster    {"namespace": "clusters", "name": "hosted-0"}
2024-02-22T09:56:19-05:00    ERROR    HostedCluster deletion failed    {"namespace": "clusters", "name": "hosted-0", "error": "context deadline exceeded"}
github.com/openshift/hypershift/cmd/cluster/core.waitForClusterDeletion
    /home/kni/hypershift_working/hypershift/cmd/cluster/core/destroy.go:268
github.com/openshift/hypershift/cmd/cluster/core.DestroyCluster
    /home/kni/hypershift_working/hypershift/cmd/cluster/core/destroy.go:130
github.com/openshift/hypershift/cmd/cluster/none.DestroyCluster
    /home/kni/hypershift_working/hypershift/cmd/cluster/none/destroy.go:47
github.com/openshift/hypershift/cmd/cluster/agent.DestroyCluster
    /home/kni/hypershift_working/hypershift/cmd/cluster/agent/destroy.go:40
github.com/openshift/hypershift/product-cli/cmd/cluster/agent.NewDestroyCommand.func1
    /home/kni/hypershift_working/hypershift/product-cli/cmd/cluster/agent/destroy.go:19
github.com/spf13/cobra.(*Command).execute
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:1115
github.com/spf13/cobra.(*Command).Execute
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:1032
main.main
    /home/kni/hypershift_working/hypershift/product-cli/main.go:60
runtime.main
    /home/kni/hypershift_working/go/src/runtime/proc.go:267
2024-02-22T09:56:19-05:00    ERROR    Failed to destroy cluster    {"error": "context deadline exceeded"}
github.com/openshift/hypershift/product-cli/cmd/cluster/agent.NewDestroyCommand.func1
    /home/kni/hypershift_working/hypershift/product-cli/cmd/cluster/agent/destroy.go:20
github.com/spf13/cobra.(*Command).execute
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:1115
github.com/spf13/cobra.(*Command).Execute
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext
    /home/kni/hypershift_working/hypershift/vendor/github.com/spf13/cobra/command.go:1032
main.main
    /home/kni/hypershift_working/hypershift/product-cli/main.go:60
runtime.main
    /home/kni/hypershift_working/go/src/runtime/proc.go:267
Error: context deadline exceeded
context deadline exceeded

Hosted cluster is successfully destroyed