HCP cluster backplanep05ue1 is running into OAuth issues and is unable to sync OAuth resources like IDP from OCM to the cluster, causing authentication issues like logging into the cluster.

4.14.8

Issue seems to be limited to backplanep05ue1

    1. 
    2.
    3.
    

backplanep05ue1 is a new backplane shard in the production environment and was rolled out by app-sre via app-interface. SRE attempted to debug the cluster in OHSS-31684 but we have been unable to find the root cause.

A potential lead is an issue with the external dns operator running on the corresponding management cluster as there are error with the vpc endpoint of backplanep05ue1 

$ oc-dtlogs external-dns-789fb6986d-v8jdv -n hypershift | grep vpce-svc-0db8f91759820b358

time="2024-02-15T11:04:28Z" level=warning msg="Could not find canonical hosted zone for domain vpce-019c21dc25060762a-whpxtgvv.vpce-svc-0db8f91759820b358.us-east-1.vpce.amazonaws.com. This may be because your region is not supported yet."
(repeated)