The successful uninstall of a hosted cluster can be prevented by a customer's PersistentVolume that fails to clean up. This specific case happened in production.

This is a guess, but it should be fairly reproducible.

1. Install some external CSI operator/StorageClass, like the EFS operator in this case
2. Create a PersistentVolume using that StorageClass
3. Uninstall the cluster, which may not cleanup the PersistentVolume

bash-4.4$ k get pv
NAME                                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                 STORAGECLASS   REASON   AGE
pv-openshift-operators-pvc-dev-pep   1Gi        RWX            Retain           Released   openshift-operators/pvc-pvc-dev-pep   efs-sc                  4d3h

YAML attached below in the comments - but there is no deletionTimestamp. After manually deleting the PV, the uninstall continued.

In the spirit of previous, similar OCPBUGS like https://issues.redhat.com/browse/OCPBUGS-11930, the uninstall should be able to give up on the complete cleanup of a PersistentVolume and progress after some time with some indication.

HCCO Logs looping with the following contents, specifically

{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Ensuring persistent volumes are removed","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}

Full loop:
❮ k logs -n ocm-production-243jo26a5v6i11gbqqp1a37hdslcp6dg-dev hosted-cluster-config-operator-6bbdd9cc4f-p8kqv --tail 20 -f
Defaulted container "hosted-cluster-config-operator" out of: hosted-cluster-config-operator, availability-prober (init)
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Registry operator management state is blank or removed, done cleaning up","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Image registry is removed","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Ensuring ingress controllers are removed","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"There are no ingresscontrollers, nothing to do","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Ingress controllers are removed","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Ensuring load balancers are removed","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Load balancers are removed","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Ensuring persistent volumes are removed","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"68967f3b-7f23-47f4-b547-8844e8bdc044"}
{"level":"info","ts":"2023-06-05T22:19:27Z","msg":"Reconciling","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"5c52d2e2-c700-44b6-968e-531eb641626d"}