PROW CI - Hypershift pods are not coming up during hypershift-mce-install step

OCP 4.21

Always

1.Trigger rehearsel for any job that consume hypershift-mce-install stage with 2.11 MCE version   
    

+ oc get pods -n hypershift
+ grep 'operator.*Running'
No resources found in hypershift namespace.

No pods coming up in hypershift namespace.

+ oc get pods -n hypershift
+ grep -q 'operator.*Running'
+ _hypershiftReady=1

Pods coming up in hypershift namespace.

During CI execution, the pods in the hypershift namespace is not initialising when we try to test with 2.11 MCE and 4.21 OCP. On debugging we found that the hypershift-install-job-xxx from the open-cluster-management-agent-addon namespace is in error state.

Reason: The two CRDs ipaddressclaims.ipam.cluster.x-k8s.io , ipaddresses.ipam.cluster.x-k8s.io getting generated have the version mismatch as spec.versions:- v1 & status.storedVersions:- v1beta2 . This causes Kubernetes to reject CRD reconciliation, which in turn prevents the Hypershift operator from coming up in the hypershift namespace.

Workaround: The workaround which we did to delete these problematic CRDs and the pods & job in open-cluster-management-agent-addon , this forces the CRDs to be regenerated, and after regeneration the versions match correctly (v1beta1- v1beta2), allowing Hypershift to start.