Description of problem:

HostedCluster reconciliation might fail to remove referenced resource annotations: When a hosted cluster is deleted, there is a race condition between the deletion of the HostedCluster and HostedControlPlane resources. If the hcp gets deleted first, the referenced-resource.hypershift.openshift.io/... annotations pointing to the hosted cluster being deleted will remain on the referenced resource.

The continuously growing list of annotations can eventually cause failures due to metadata.annotations: Too long: must have at most 262144 bytes errors.

Version-Release number of selected component (if applicable):

HyperShift Operator 4.20, 4.21, 4.22

Steps to Reproduce:

1. Create many hosted clusters that reference a secret or configmap.
2. Delete the hosted clusters.
    

Actual results:

Referenced resource annotations of deleted hosted clusters remain on the referenced resource.

Expected results:

Referenced resource annotations are always removed when the referencing hosted cluster is deleted.

Additional info: