-
Bug
-
Resolution: Duplicate
-
Undefined
-
None
-
4.14, 4.14.z, 4.15
-
None
-
No
-
False
-
Description of problem:
The test case https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-49750 was created for https://bugzilla.redhat.com/show_bug.cgi?id=2025624 bug. We are deleting the default CA certificate and rechecking the metric to confirm the CA certificate reloaded. The CA certificate is reloaded, but the console,kube-scheduler and monitoring are degraded. Can see from the router pod 'TLS handshake error from 10.131.0.17:60560: remote error: tls: bad certificate'
Version-Release number of selected component (if applicable):
How reproducible:
https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-49750
Steps to Reproduce:
1. Delete the default CA certificate
oc delete secret/signing-key -n openshift-service-ca
secret "signing-key" deleted
2. Check the operator status and we can some are degraded for long time
melvinjoseph@mjoseph-mac openshift-tests-private % oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.15.0-0.nightly-2023-10-16-231617 True False False 74m
baremetal 4.15.0-0.nightly-2023-10-16-231617 True False False 6h24m
cloud-controller-manager 4.15.0-0.nightly-2023-10-16-231617 True False False 6h27m
cloud-credential 4.15.0-0.nightly-2023-10-16-231617 True False False 6h29m
cluster-autoscaler 4.15.0-0.nightly-2023-10-16-231617 True False False 6h24m
config-operator 4.15.0-0.nightly-2023-10-16-231617 True False False 6h25m
console 4.15.0-0.nightly-2023-10-16-231617 False False True 108m RouteHealthAvailable: route not yet available, https://console-openshift-console.apps.mjoseph-bugazure.qe.azure.devcluster.openshift.com returns '503 Service Unavailable'
control-plane-machine-set 4.15.0-0.nightly-2023-10-16-231617 True False False 97m
csi-snapshot-controller 4.15.0-0.nightly-2023-10-16-231617 True False False 6h9m
dns 4.15.0-0.nightly-2023-10-16-231617 True False False 6h24m
etcd 4.15.0-0.nightly-2023-10-16-231617 True False False 6h16m
image-registry 4.15.0-0.nightly-2023-10-16-231617 True False False 5h25m
ingress 4.15.0-0.nightly-2023-10-16-231617 True False False 6h8m
insights 4.15.0-0.nightly-2023-10-16-231617 True False False 6h19m
kube-apiserver 4.15.0-0.nightly-2023-10-16-231617 True False False 6h14m
kube-controller-manager 4.15.0-0.nightly-2023-10-16-231617 True False True 6h15m GarbageCollectorDegraded: alerts firing: GarbageCollectorSyncFailed
kube-scheduler 4.15.0-0.nightly-2023-10-16-231617 True False False 6h16m
kube-storage-version-migrator 4.15.0-0.nightly-2023-10-16-231617 True False False 6h25m
machine-api 4.15.0-0.nightly-2023-10-16-231617 True False False 6h14m
machine-approver 4.15.0-0.nightly-2023-10-16-231617 True False False 6h25m
machine-config 4.15.0-0.nightly-2023-10-16-231617 True False False 6h22m
marketplace 4.15.0-0.nightly-2023-10-16-231617 True False False 6h24m
monitoring 4.15.0-0.nightly-2023-10-16-231617 False True True 74m reconciling Console Plugin failed: retrieving ConsolePlugin object failed: conversion webhook for console.openshift.io/v1alpha1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
network 4.15.0-0.nightly-2023-10-16-231617 True False False 6h27m
node-tuning 4.15.0-0.nightly-2023-10-16-231617 True False False 6h9m
openshift-apiserver 4.15.0-0.nightly-2023-10-16-231617 True False False 74m
openshift-controller-manager 4.15.0-0.nightly-2023-10-16-231617 True False False 6h9m
openshift-samples 4.15.0-0.nightly-2023-10-16-231617 True False False 6h9m
operator-lifecycle-manager 4.15.0-0.nightly-2023-10-16-231617 True False False 6h24m
operator-lifecycle-manager-catalog 4.15.0-0.nightly-2023-10-16-231617 True False False 6h24m
operator-lifecycle-manager-packageserver 4.15.0-0.nightly-2023-10-16-231617 True False False 6h13m
service-ca 4.15.0-0.nightly-2023-10-16-231617 True False False 6h25m
storage 4.15.0-0.nightly-2023-10-16-231617 True False False 6h6m
Actual results:
console,kube-scheduler and monitoring are degraded
Expected results:
All operator should be working fine.
Additional info:
melvinjoseph@mjoseph-mac openshift-tests-private % oc get pod -n openshift-ingress NAME READY STATUS RESTARTS AGE router-default-7dcd556587-hfjcx 1/1 Running 0 135m router-default-7dcd556587-vppk4 1/1 Running 0 6h32m melvinjoseph@mjoseph-mac openshift-tests-private % oc logs -n openshift-ingress router-default-7dcd556587-hfjcx I1017 11:58:51.827625 1 template.go:559] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: f142a3a4f2890527d6b22c211faf04f34ed86625\nversionFromGit: 4.0.0-464-gf142a3a4\ngitTreeState: clean\nbuildDate: 2023-10-11T15:33:53Z\n" I1017 11:58:51.829764 1 metrics.go:156] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936" I1017 11:58:51.835625 1 router.go:210] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy" I1017 11:58:51.835712 1 router.go:294] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s" I1017 11:58:51.836241 1 router.go:364] template "msg"="watching for changes" "path"="/etc/pki/tls/private" I1017 11:58:51.836311 1 router.go:269] router "msg"="router is including routes in all namespaces" E1017 11:58:51.947702 1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory I1017 11:58:51.991562 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:14:03.116934 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:14:35.654294 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:14:40.665442 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:14:51.830865 1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key" I1017 12:15:08.457440 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:15:13.453119 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:15:43.677935 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:28:37.854556 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:28:42.846824 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:28:47.847065 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:50:27.445556 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:50:53 http: TLS handshake error from 10.131.0.17:47956: remote error: tls: bad certificate I1017 12:50:58.620930 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:51:03.623353 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:51:23 http: TLS handshake error from 10.131.0.17:35550: remote error: tls: bad certificate I1017 12:51:27.763545 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:51:32.760809 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:51:41 http: TLS handshake error from 10.128.2.19:43202: remote error: tls: bad certificate 2023/10/17 12:51:53 http: TLS handshake error from 10.131.0.17:45296: remote error: tls: bad certificate I1017 12:52:02.595749 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:52:11 http: TLS handshake error from 10.128.2.19:37530: remote error: tls: bad certificate I1017 12:52:21.831016 1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key" melvinjoseph@mjoseph-mac openshift-tests-private % melvinjoseph@mjoseph-mac openshift-tests-private % oc logs -n openshift-ingress router-default-7dcd556587-vppk4 I1017 07:53:40.093384 1 template.go:559] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: f142a3a4f2890527d6b22c211faf04f34ed86625\nversionFromGit: 4.0.0-464-gf142a3a4\ngitTreeState: clean\nbuildDate: 2023-10-11T15:33:53Z\n" I1017 07:53:40.095729 1 metrics.go:156] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936" I1017 07:53:40.101732 1 router.go:210] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy" I1017 07:53:40.101845 1 router.go:294] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s" I1017 07:53:40.103532 1 router.go:364] template "msg"="watching for changes" "path"="/etc/pki/tls/private" I1017 07:53:40.103659 1 router.go:269] router "msg"="router is including routes in all namespaces" E1017 07:53:40.208831 1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory I1017 07:53:40.272801 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:53:47.896871 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:53:52.433465 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:54:02.819311 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:54:07.818245 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:55:51.291930 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:55:56.287743 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:56:55.371021 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:57:00.833787 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 07:57:05.826005 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:00:08.932683 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:00:13.933772 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:00:33.344017 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:00:38.307033 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:00:45.032138 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:00:50.372577 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:00:55.652090 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:01:18.332044 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:02:15.357935 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:02:22.076826 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:02:45.844840 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:09:34.489414 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:10:02.009849 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:10:06.993218 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:10:29.802282 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:10:34.743693 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:10:59.602493 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:36:13.335481 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:36:18.333081 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:37:32.783100 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 08:37:37.764976 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:14:03.122601 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:14:35.736461 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:14:40.676549 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:15:01 http: TLS handshake error from 10.131.0.17:59370: remote error: tls: bad certificate I1017 12:15:08.453402 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:15:13.457828 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:15:24 http: TLS handshake error from 10.128.2.19:33956: remote error: tls: bad certificate 2023/10/17 12:15:31 http: TLS handshake error from 10.131.0.17:60560: remote error: tls: bad certificate I1017 12:15:40.096326 1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key" I1017 12:15:43.681101 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:28:37.860814 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:28:42.848445 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:28:47.858610 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:50:27.437783 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:50:58.629797 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:51:01 http: TLS handshake error from 10.131.0.17:39122: remote error: tls: bad certificate I1017 12:51:03.633762 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:51:24 http: TLS handshake error from 10.128.2.19:36280: remote error: tls: bad certificate I1017 12:51:27.780906 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:51:31 http: TLS handshake error from 10.131.0.17:54172: remote error: tls: bad certificate I1017 12:51:32.769168 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2023/10/17 12:51:54 http: TLS handshake error from 10.128.2.19:56850: remote error: tls: bad certificate 2023/10/17 12:52:01 http: TLS handshake error from 10.131.0.17:52022: remote error: tls: bad certificate I1017 12:52:02.600832 1 router.go:649] template "msg"="router reloaded" "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I1017 12:52:10.096182 1 template.go:925] router "msg"="reloaded metrics certificate" "cert"="/etc/pki/tls/metrics-certs/tls.crt" "key"="/etc/pki/tls/metrics-certs/tls.key"
- duplicates
-
-
- Closed
-
