The client cert/key pair to authenticate the function even without live kube-apiserver connections is causing TLS handshake error. Its unable to verify the certificate. The endpoints are rejecting the TLS certificate from Prometheus.

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-gcp-ovn-upgrade/1643172085851230208/artifacts/e2e-gcp-ovn-upgrade/gather-extra/artifacts/metrics/prometheus-targets.json  | jq '.data.activeTargets | map(select(.health=="down")) | map({"lastError": .lastError,"scrapePool":.scrapePool})'
[
  {
    "lastError": "Get \"https://10.130.0.39:8443/metrics\": remote error: tls: bad certificate",
    "scrapePool": "serviceMonitor/openshift-operator-lifecycle-manager/catalog-operator/0"
  },
  {
    "lastError": "Get \"https://10.130.0.40:8443/metrics\": remote error: tls: bad certificate",
    "scrapePool": "serviceMonitor/openshift-operator-lifecycle-manager/olm-operator/0"
  }
]

https://github.com/openshift/operator-framework-olm/pull/368 


https://grafana-loki.ci.openshift.org/explore?orgId=1&left=%7B%22datasource%22:%22PCB22D447805DBCCF%22,%22queries%22:%5B%7B%22expr%22:%22%7Binvoker%3D%5C%22openshift-internal-ci%2Fperiodic-ci-openshift-release-master-ci-4.14-e2e-gcp-ovn-upgrade%2F1643172085851230208%5C%22%7D%20%7C%20unpack%20%7C%20namespace%3D%5C%22openshift-operator-lifecycle-manager%5C%22%20%20%7C~%5C%22tls%5C%22%22,%22refId%22:%22A%22,%22editorMode%22:%22code%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D

1.
2.
3.

2023-04-04 07:28:282023/04/04 11:28:28 http: TLS handshake error from 10.129.2.5:33368: tls: failed to verify client certificate: x509: certificate signed by unknown authority