The test case https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-49750 was created for https://bugzilla.redhat.com/show_bug.cgi?id=2025624 bug. We are deleting the default CA certificate and rechecking the metric to confirm the CA certificate reloaded. The CA certificate is reloaded, but the console,kube-scheduler and monitoring are degraded.  Can see from the router pod 'TLS handshake error from 10.131.0.17:60560: remote error: tls: bad certificate'

https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-49750

1. Delete the default CA certificate
    oc delete secret/signing-key -n openshift-service-ca 
    secret "signing-key" deleted
2. Check the operator status and we can some are degraded for long time
   melvinjoseph@mjoseph-mac openshift-tests-private % oc get co 
NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.nightly-2023-10-16-231617   True        False         False      74m     
baremetal                                  4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h24m   
cloud-controller-manager                   4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h27m   
cloud-credential                           4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h29m   
cluster-autoscaler                         4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h24m   
config-operator                            4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h25m   
console                                    4.15.0-0.nightly-2023-10-16-231617   False       False         True       108m    RouteHealthAvailable: route not yet available, https://console-openshift-console.apps.mjoseph-bugazure.qe.azure.devcluster.openshift.com returns '503 Service Unavailable'
control-plane-machine-set                  4.15.0-0.nightly-2023-10-16-231617   True        False         False      97m     
csi-snapshot-controller                    4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h9m    
dns                                        4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h24m   
etcd                                       4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h16m   
image-registry                             4.15.0-0.nightly-2023-10-16-231617   True        False         False      5h25m   
ingress                                    4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h8m    
insights                                   4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h19m   
kube-apiserver                             4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h14m   
kube-controller-manager                    4.15.0-0.nightly-2023-10-16-231617   True        False         True       6h15m   GarbageCollectorDegraded: alerts firing: GarbageCollectorSyncFailed
kube-scheduler                             4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h16m   
kube-storage-version-migrator              4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h25m   
machine-api                                4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h14m   
machine-approver                           4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h25m   
machine-config                             4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h22m   
marketplace                                4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h24m   
monitoring                                 4.15.0-0.nightly-2023-10-16-231617   False       True          True       74m     reconciling Console Plugin failed: retrieving ConsolePlugin object failed: conversion webhook for console.openshift.io/v1alpha1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
network                                    4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h27m   
node-tuning                                4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h9m    
openshift-apiserver                        4.15.0-0.nightly-2023-10-16-231617   True        False         False      74m     
openshift-controller-manager               4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h9m    
openshift-samples                          4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h9m    
operator-lifecycle-manager                 4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h24m   
operator-lifecycle-manager-catalog         4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h24m   
operator-lifecycle-manager-packageserver   4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h13m   
service-ca                                 4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h25m   
storage                                    4.15.0-0.nightly-2023-10-16-231617   True        False         False      6h6m 

console,kube-scheduler and monitoring are degraded

All operator should be working fine.

melvinjoseph@mjoseph-mac openshift-tests-private % oc get pod -n openshift-ingress
NAME                              READY   STATUS    RESTARTS   AGE
router-default-7dcd556587-hfjcx   1/1     Running   0          135m
router-default-7dcd556587-vppk4   1/1     Running   0          6h32m
melvinjoseph@mjoseph-mac openshift-tests-private % oc logs -n openshift-ingress router-default-7dcd556587-hfjcx
I1017 11:58:51.827625       1 template.go:559] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: f142a3a4f2890527d6b22c211faf04f34ed86625\nversionFromGit: 4.0.0-464-gf142a3a4\ngitTreeState: clean\nbuildDate: 2023-10-11T15:33:53Z\n"
I1017 11:58:51.829764       1 metrics.go:156] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936"
I1017 11:58:51.835625       1 router.go:210] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy"
I1017 11:58:51.835712       1 router.go:294] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s"
I1017 11:58:51.836241       1 router.go:364] template "msg"="watching for changes" "path"="/etc/pki/tls/private"
I1017 11:58:51.836311       1 router.go:269] router "msg"="router is including routes in all namespaces" 
E1017 11:58:51.947702       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
I1017 11:58:51.991562       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:14:03.116934       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:14:35.654294       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:14:40.665442       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:14:51.830865       1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key"
I1017 12:15:08.457440       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:15:13.453119       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:15:43.677935       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:28:37.854556       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:28:42.846824       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:28:47.847065       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:50:27.445556       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:50:53 http: TLS handshake error from 10.131.0.17:47956: remote error: tls: bad certificate
I1017 12:50:58.620930       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:51:03.623353       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:51:23 http: TLS handshake error from 10.131.0.17:35550: remote error: tls: bad certificate
I1017 12:51:27.763545       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:51:32.760809       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:51:41 http: TLS handshake error from 10.128.2.19:43202: remote error: tls: bad certificate
2023/10/17 12:51:53 http: TLS handshake error from 10.131.0.17:45296: remote error: tls: bad certificate
I1017 12:52:02.595749       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:52:11 http: TLS handshake error from 10.128.2.19:37530: remote error: tls: bad certificate
I1017 12:52:21.831016       1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key"
melvinjoseph@mjoseph-mac openshift-tests-private % 



melvinjoseph@mjoseph-mac openshift-tests-private % oc logs -n openshift-ingress router-default-7dcd556587-vppk4
I1017 07:53:40.093384       1 template.go:559] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: f142a3a4f2890527d6b22c211faf04f34ed86625\nversionFromGit: 4.0.0-464-gf142a3a4\ngitTreeState: clean\nbuildDate: 2023-10-11T15:33:53Z\n"
I1017 07:53:40.095729       1 metrics.go:156] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936"
I1017 07:53:40.101732       1 router.go:210] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy"
I1017 07:53:40.101845       1 router.go:294] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s"
I1017 07:53:40.103532       1 router.go:364] template "msg"="watching for changes" "path"="/etc/pki/tls/private"
I1017 07:53:40.103659       1 router.go:269] router "msg"="router is including routes in all namespaces" 
E1017 07:53:40.208831       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
I1017 07:53:40.272801       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:53:47.896871       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:53:52.433465       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:54:02.819311       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:54:07.818245       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:55:51.291930       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:55:56.287743       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:56:55.371021       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:57:00.833787       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 07:57:05.826005       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:00:08.932683       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:00:13.933772       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:00:33.344017       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:00:38.307033       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:00:45.032138       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:00:50.372577       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:00:55.652090       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:01:18.332044       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:02:15.357935       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:02:22.076826       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:02:45.844840       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:09:34.489414       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:10:02.009849       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:10:06.993218       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:10:29.802282       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:10:34.743693       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:10:59.602493       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:36:13.335481       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:36:18.333081       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:37:32.783100       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 08:37:37.764976       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:14:03.122601       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:14:35.736461       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:14:40.676549       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:15:01 http: TLS handshake error from 10.131.0.17:59370: remote error: tls: bad certificate
I1017 12:15:08.453402       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:15:13.457828       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:15:24 http: TLS handshake error from 10.128.2.19:33956: remote error: tls: bad certificate
2023/10/17 12:15:31 http: TLS handshake error from 10.131.0.17:60560: remote error: tls: bad certificate
I1017 12:15:40.096326       1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key"
I1017 12:15:43.681101       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:28:37.860814       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:28:42.848445       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:28:47.858610       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:50:27.437783       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:50:58.629797       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:51:01 http: TLS handshake error from 10.131.0.17:39122: remote error: tls: bad certificate
I1017 12:51:03.633762       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:51:24 http: TLS handshake error from 10.128.2.19:36280: remote error: tls: bad certificate
I1017 12:51:27.780906       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:51:31 http: TLS handshake error from 10.131.0.17:54172: remote error: tls: bad certificate
I1017 12:51:32.769168       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
2023/10/17 12:51:54 http: TLS handshake error from 10.128.2.19:56850: remote error: tls: bad certificate
2023/10/17 12:52:01 http: TLS handshake error from 10.131.0.17:52022: remote error: tls: bad certificate
I1017 12:52:02.600832       1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I1017 12:52:10.096182       1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key"