Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18268

cluster-node-tuning-operator throws TLS handshake error remote error: tls: bad certificate

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • No
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Issue:

      cluster-node-tuning-operator throws TLS handshake error remote error: tls: bad certificate

      I am able to reproduce the issue the customer mentioned in my test cluster

      Below are the testing Results from my test cluster 4.12.7

      ~~~

      [sasakshi@sasakshi ~]$ oc version
      Client Version: 4.12.7
      Kustomize Version: v4.5.7
      Server Version: 4.12.22
      Kubernetes Version: v1.25.10+8c21020
      [sasakshi@sasakshi ~]$
      
      [sasakshi@sasakshi ~]$ oc project openshift-cluster-node-tuning-operator
      Now using project "openshift-cluster-node-tuning-operator" on server "https://api.saktest412.lab.upshift.rdu2.redhat.com:6443".
      [sasakshi@sasakshi ~]$ oc get pods
      NAME READY STATUS RESTARTS AGE
      cluster-node-tuning-operator-87c489b45-b8w85 1/1 Running 1 (36d ago) 36d
      tuned-6j68j 1/1 Running 0 36d
      tuned-dbxg5 1/1 Running 0 36d
      tuned-fnddt 1/1 Running 0 36d
      
      [sasakshi@sasakshi ~]$ oc logs cluster-node-tuning-operator-87c489b45-b8w85 | grep -i "error" | tail -10
      2023/08/24 00:15:32 http: TLS handshake error from 10.129.0.3:41684: remote error: tls: bad certificate
      2023/08/24 00:54:09 http: TLS handshake error from 10.129.0.3:33872: remote error: tls: bad certificate
      2023/08/24 00:54:10 http: TLS handshake error from 10.129.0.3:33882: remote error: tls: bad certificate
      2023/08/24 00:54:12 http: TLS handshake error from 10.129.0.3:45972: remote error: tls: bad certificate
      2023/08/24 05:07:35 http: TLS handshake error from 10.129.0.3:33912: remote error: tls: bad certificate
      2023/08/24 05:07:36 http: TLS handshake error from 10.129.0.3:33920: remote error: tls: bad certificate
      2023/08/24 05:07:38 http: TLS handshake error from 10.129.0.3:33928: remote error: tls: bad certificate
      2023/08/24 06:03:25 http: TLS handshake error from 10.129.0.3:34384: remote error: tls: bad certificate
      2023/08/24 06:03:26 http: TLS handshake error from 10.129.0.3:34392: remote error: tls: bad certificate
      2023/08/24 06:03:28 http: TLS handshake error from 10.129.0.3:34404: remote error: tls: bad certificate
      
      [sasakshi@sasakshi ~]$ oc get pods -o wide -A | grep "10.129.0.3"
      
      openshift-kube-apiserver-operator kube-apiserver-operator-6ffd6b76ff-9wwvq 1/1 Running 3 (39d ago) 39d 10.129.0.3 master-2.saktest412.lab.upshift.rdu2.redhat.com <none> <none>
      
      sasakshi@sasakshi ~]$ oc logs kube-apiserver-operator-6ffd6b76ff-9wwvq -n openshift-kube-apiserver-operator | grep -A2 "unknown" | tail -20
      
      I0725 21:06:33.594131 1 request.go:601] Waited for 1.56492663s due to client-side throttling, not priority and fairness, request: GET:[https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/client-ca|https://172.30.0.1/api/v1/namespaces/openshift-kube-apiserver/configmaps/client-ca]
      E0725 21:10:30.918857 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      E0725 21:10:31.924119 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      I0725 21:10:35.050202 1 request.go:601] Waited for 1.077837858s due to client-side throttling, not priority and fairness, request: GET:[https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver|https://172.30.0.1/api/v1/namespaces/openshift-kube-apiserver]
      I0725 21:10:36.052584 1 request.go:601] Waited for 1.591518505s due to client-side throttling, not priority and fairness, request: GET:[https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs|https://172.30.0.1/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs]
      –
      E0725 21:10:45.437433 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      E0725 21:10:46.442653 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      E0725 21:10:48.514765 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      E0725 21:10:49.520979 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      I0725 21:10:52.749883 1 request.go:601] Waited for 1.103225368s due to client-side throttling, not priority and fairness, request: GET:[https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/serviceaccounts/installer-sa|https://172.30.0.1/api/v1/namespaces/openshift-kube-apiserver/serviceaccounts/installer-sa]
      I0725 21:10:53.949479 1 request.go:601] Waited for 1.5977259s due to client-side throttling, not priority and fairness, request: GET:[https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/client-ca|https://172.30.0.1/api/v1/namespaces/openshift-kube-apiserver/configmaps/client-ca]
      –
      E0725 21:14:48.784832 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      E0725 21:14:49.790257 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
      I0725 21:14:53.029390 1 request.go:601] Waited for 1.099550177s due to client-side throttling, not priority and fairness, request: GET:[https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/kube-apiserver-audit-policies|https://172.30.0.1/api/v1/namespaces/openshift-kube-apiserver/configmaps/kube-apiserver-audit-policies]
      I0725 21:14:54.229868 1 request.go:601] Waited for 1.59697648s due to client-side throttling, not priority and fairness, request: GET:[https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/revision-pruner-13-master-2.saktest412.lab.upshift.rdu2.redhat.com|https://172.30.0.1/api/v1/namespaces/openshift-kube-apiserver/pods/revision-pruner-13-master-2.saktest412.lab.upshift.rdu2.redhat.com]
      
      sasakshi@sasakshi ~]$ oc debug cluster-node-tuning-operator-87c489b45-b8w85
      Starting pod/cluster-node-tuning-operator-87c489b45-b8w85-debug, command was: cluster-node-tuning-operator -v=0
      Pod IP: 10.129.1.194
      If you don't see a command prompt, try pressing enter.
      sh-4.4$ curl -vvv -k 10.129.0.3:41684
      
      Rebuilt URL to: 10.129.0.3:41684/
      
      Trying 10.129.0.3...
      
      TCP_NODELAY set
      
      connect to 10.129.0.3 port 41684 failed: Connection refused
      
      Failed to connect to 10.129.0.3 port 41684: Connection refused
      
      Closing connection 0
      curl: (7) Failed to connect to 10.129.0.3 port 41684: Connection refused
      

      ~~~

              Unassigned Unassigned
              sasakshi@redhat.com Sakshi sakshi
              None
              None
              Mallapadi Niranjan Mallapadi Niranjan
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: