Description of problem:

The production ROSA HCP clusters (IBM 4.13) are experiencing connectivity issues. Not able to connect to Production cluster ROSA IBM 4.13 and 4.14 clusters not able to connect to console

Summarizing the current updates based on our SRE's investigation so far: 

• Our SRE has done some investigation and looks it primarily due to expired certificates affecting konnectivity-agent DS pods. Although the certificate secrets have been updated, there seems to be a propagation issue.
• Openshift-apiserver pods also show authorization errors, with older pods affected more than newer ones, possibly due to the same certificate issues.
• Additionally, console pods are facing authorization issues linked to OAuth problems, which in turn are likely caused by communication issues with the API server.

Three clusters potentially impacted - documented in two tickets:

• https://issues.redhat.com/browse/OHSS-35643
• https://issues.redhat.com/browse/OHSS-35563

More discussions in the itn-2024-00107

Version-Release number of selected component (if applicable):

4.13.13

How reproducible:

The clusters lost access and customer cannot log in nether using OpenID or htpasswd