-
Bug
-
Resolution: Cannot Reproduce
-
Undefined
-
None
-
4.19.0
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Starting with Networking / router but reassign as appropriate.
We have detected a large spike in vsphere disruption recently
Reviewing periodic-ci-openshift-release-master-ci-4.19-upgrade-from-stable-4.18-e2e-vsphere-ovn-upgrade/1894941148963672064 show that on the 3rd master node update (master-0) we can see disruption far outside the norm
We expect master-0 to be unavailable during the upgrade but both master-1 and 2 show issues like
Feb 27 04:37:12.513718 ci-op-js44mi3w-c9cbb-xqrlk-master-1 kubenswrapper[2748]: I0227 04:37:12.513700 2748 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF Feb 27 04:37:12.513762 ci-op-js44mi3w-c9cbb-xqrlk-master-1 kubenswrapper[2748]: I0227 04:37:12.513756 2748 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF Feb 27 04:37:12.513836 ci-op-js44mi3w-c9cbb-xqrlk-master-1 kubenswrapper[2748]: I0227 04:37:12.513810 2748 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF Feb 27 04:37:15.840207 ci-op-js44mi3w-c9cbb-xqrlk-master-1 kubenswrapper[2748]: I0227 04:37:15.840170 2748 ???:1] "http: TLS handshake error from 10.93.251.91:59764: EOF" Feb 27 04:37:20.840601 ci-op-js44mi3w-c9cbb-xqrlk-master-1 kubenswrapper[2748]: I0227 04:37:20.840568 2748 ???:1] "http: TLS handshake error from 10.93.251.91:59816: EOF" Feb 27 04:37:25.841147 ci-op-js44mi3w-c9cbb-xqrlk-master-1 kubenswrapper[2748]: I0227 04:37:25.841106 2748 ???:1] "http: TLS handshake error from 10.93.251.91:43462: EOF"
Feb 27 04:37:12.513704 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:12.513306 2853 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF Feb 27 04:37:12.513704 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:12.513355 2853 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF Feb 27 04:37:12.513704 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:12.513366 2853 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF Feb 27 04:37:13.117079 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:13.117034 2853 webhook.go:269] Failed to make webhook authorizer request: Post "https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews": dial tcp 10.93.251.16:6443: connect: connection refused Feb 27 04:37:13.117079 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:13.117072 2853 server.go:339] "Authorization error" err="Post \"https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp 10.93.251.16:6443: connect: connection refused" user="system:serviceaccount:openshift-monitoring:prometheus-k8s" verb="get" resource="nodes" subresource="metrics" Feb 27 04:37:13.205469 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:13.205432 2853 webhook.go:269] Failed to make webhook authorizer request: Post "https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews": dial tcp 10.93.251.16:6443: connect: connection refused Feb 27 04:37:13.205469 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:13.205465 2853 server.go:339] "Authorization error" err="Post \"https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp 10.93.251.16:6443: connect: connection refused" user="system:serviceaccount:openshift-monitoring:prometheus-k8s" verb="get" resource="nodes" subresource="metrics" Feb 27 04:37:14.848683 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:14.848646 2853 webhook.go:269] Failed to make webhook authorizer request: Post "https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews": dial tcp 10.93.251.16:6443: connect: connection refused Feb 27 04:37:14.848683 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:14.848679 2853 server.go:339] "Authorization error" err="Post \"https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp 10.93.251.16:6443: connect: connection refused" user="system:serviceaccount:openshift-monitoring:prometheus-k8s" verb="get" resource="nodes" subresource="metrics" Feb 27 04:37:16.510188 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:16.510144 2853 ???:1] "http: TLS handshake error from 10.93.251.64:54846: EOF" Feb 27 04:37:16.727311 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:16.727275 2853 webhook.go:269] Failed to make webhook authorizer request: Post "https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews": dial tcp 10.93.251.16:6443: connect: connection refused Feb 27 04:37:16.727311 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: E0227 04:37:16.727305 2853 server.go:339] "Authorization error" err="Post \"https://api-int.ci-op-js44mi3w-c9cbb.vmc-ci.devcluster.openshift.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp 10.93.251.16:6443: connect: connection refused" user="system:serviceaccount:openshift-monitoring:prometheus-k8s" verb="get" resource="nodes" subresource="metrics" Feb 27 04:37:21.509542 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:21.509502 2853 ???:1] "http: TLS handshake error from 10.93.251.64:37138: EOF" Feb 27 04:37:26.510226 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:26.510076 2853 ???:1] "http: TLS handshake error from 10.93.251.64:37180: EOF" Feb 27 04:37:31.509480 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:31.509441 2853 ???:1] "http: TLS handshake error from 10.93.251.64:48868: EOF" Feb 27 04:37:36.509402 ci-op-js44mi3w-c9cbb-xqrlk-master-2 kubenswrapper[2853]: I0227 04:37:36.509362 2853 ???:1] "http: TLS handshake error from 10.93.251.64:48936: EOF"
