Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4999

[OpenShiftSDN ] API svc un-reachability on OCP upgrade 4.10->4.11

XMLWordPrintable

    • None
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      During an upgrade from 4.10.0-0.nightly-2022-09-06-081345 -> 4.11.0-0.nightly-2022-09-06-074353

DNS operator stuck in progressing state with following error in dns-defalt pod

2022-09-07T00:47:35.959931289Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout

must-gather: http://shell.lab.bos.redhat.com/~anusaxen/must-gather-136172-050448657.tar.gz

      Version-Release number of selected component (if applicable):

      4.11.0-0.nightly-2022-09-06-074353

      How reproducible:

      Intermittent

      Steps to Reproduce:

      1. Perform upgrade from 4.10->4.11
2.
3.

      Actual results:

      Upgrade unsuccessful due to API svc unreachability

      Expected results:

      Upgrade should be successful

      Additional info:

      $ omg get co | grep -v "True.*False.*False"
NAME                                      VERSION                             AVAILABLE  PROGRESSING  DEGRADED  SINCE
dns                                       4.10.0-0.nightly-2022-09-06-081345  True       True         False     3h4m
network                                   4.11.0-0.nightly-2022-09-06-074353  True       True         False     3h5m

$ omg logs dns-default-5mqz4 -c dns -n openshift-dns
/home/anusaxen/Downloads/must-gather-136208-318613238/must-gather.local.4033693973186012455/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-d995e1608ba83a1f71b5a1c49705aa3cef38618e0674ee256332d1b0e2cb0e23/namespaces/openshift-dns/pods/dns-default-5mqz4/dns/dns/logs/current.log
2022-09-07T00:45:36.355161941Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:36.854403897Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:37.354367197Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:37.861427861Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:38.355095996Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:38.855063626Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:39.355170472Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:39.855296624Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:40.355481413Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:40.855455283Z [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
2022-09-07T00:45:40.856166731Z .:5353
2022-09-07T00:45:40.856166731Z [INFO] plugin/reload: Running configuration SHA512 = 7c3db3182389e270fe3af971fef9e7157a34bde01eb750d58c2d4895a965ecb2e2d00df34594b6a819653088b6efa2590e748ce23f9e16cbb0f44d74a4fc3587
2022-09-07T00:45:40.856166731Z CoreDNS-1.9.2
2022-09-07T00:45:40.856166731Z linux/amd64, go1.18.4, 
2022-09-07T00:46:05.957503195Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout
2022-09-07T00:47:25.444876534Z [INFO] SIGTERM: Shutting down servers then terminating
2022-09-07T00:47:25.444984367Z [INFO] plugin/health: Going into lameduck mode for 20s
2022-09-07T00:47:35.959931289Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout

              ffernand@redhat.com Flavio Fernandes (Inactive)
              anusaxen Anurag Saxena
              Zhanqi Zhao Zhanqi Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: