Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-50498

cluster-version-operator panics due to a port collision

XMLWordPrintable

    • Important
    • None
    • 2
    • OTA 267
    • 1
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      cluster-version-operator pod is crashing repeatedly. It is a disconnected cluster. Customer used to ABI (agent based installation) and it's bare metal.  
      
      # oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.16.7    True        False         134d    Cluster version is 4.16.7
      
      $ omc get pods
      NAME                                        READY   STATUS   RESTARTS   AGE
      cluster-version-operator-74688c8bc5-fbzlf   0/1     Error    1425       7d
      
      ====
      2025-02-07T10:24:37.750428195+05:30 I0207 04:54:37.750394       1 start.go:565] FeatureGate found in cluster, using its feature set "" at startup
      2025-02-07T10:24:37.750437148+05:30 I0207 04:54:37.750427       1 payload.go:307] Loading updatepayload from "/"
      2025-02-07T10:24:37.750603865+05:30 I0207 04:54:37.750583       1 payload.go:403] Architecture from release-metadata (4.16.7) retrieved from runtime: "amd64"
      2025-02-07T10:24:37.751094450+05:30 I0207 04:54:37.751078       1 metrics.go:154] Metrics port listening for HTTPS on 0.0.0.0:9099
      2025-02-07T10:24:37.751105629+05:30 E0207 04:54:37.751091       1 metrics.go:166] Collected metrics HTTPS server goroutine: listen tcp 0.0.0.0:9099: bind: address already in use
      2025-02-07T10:24:37.754776401+05:30 E0207 04:54:37.754748       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      2025-02-07T10:24:37.754776401+05:30 goroutine 236 [running]:
      2025-02-07T10:24:37.754776401+05:30 k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1bdfc40?, 0x311dc90})
      2025-02-07T10:24:37.754776401+05:30     /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85
      2025-02-07T10:24:37.754776401+05:30 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0004fc370?})
      2025-02-07T10:24:37.754776401+05:30     /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b
      2025-02-07T10:24:37.754776401+05:30 panic({0x1bdfc40?, 0x311dc90?})
      2025-02-07T10:24:37.754776401+05:30     /usr/lib/golang/src/runtime/panic.go:914 +0x21f
      2025-02-07T10:24:37.754776401+05:30 crypto/tls.(*listener).Close(0x1e67240?)
      2025-02-07T10:24:37.754776401+05:30     <autogenerated>:1 +0x1e
      2025-02-07T10:24:37.754776401+05:30 net/http.(*onceCloseListener).close(...)
      2025-02-07T10:24:37.754776401+05:30     /usr/lib/golang/src/net/http/server.go:3502
      2025-02-07T10:24:37.754776401+05:30 sync.(*Once).doSlow(0xc000701b68?, 0x414e32?)
      2025-02-07T10:24:37.754776401+05:30     /usr/lib/golang/src/sync/once.go:74 +0xbf
      2025-02-07T10:24:37.754776401+05:30 sync.(*Once).Do(...)
      2025-02-07T10:24:37.754776401+05:30     /usr/lib/golang/src/sync/once.go:65
      2025-02-07T10:24:37.754776401+05:30 net/http.(*onceCloseListener).Close(0xc000346ab0)
      2025-02-07T10:24:37.754776401+05:30     /usr/lib/golang/src/net/http/server.go:3498 +0x45
      2025-02-07T10:24:37.754776401+05:30 net/http.(*Server).closeListenersLocked(0xa6?)
      2025-02-07T10:24:37.754776401+05:30     /usr/lib/golang/src/net/http/server.go:2864 +0x96
      2025-02-07T10:24:37.754776401+05:30 net/http.(*Server).Shutdown(0xc0006c00f0, {0x21f3160, 0xc0002c85f0})
      2025-02-07T10:24:37.754776401+05:30     /usr/lib/golang/src/net/http/server.go:2790 +0x96
      2025-02-07T10:24:37.754776401+05:30 github.com/openshift/cluster-version-operator/pkg/cvo.RunMetrics({0x21f3160, 0xc0002c8640}, {0x21f3160, 0xc0002c85f0}, {0x7ffff38f18a7, 0xc}, {0x7ffff38f18c8, 0x1d}, {0x7ffff38f18f9, 0x1d})
      2025-02-07T10:24:37.754776401+05:30     /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/metrics.go:292 +0x130c
      2025-02-07T10:24:37.754776401+05:30 github.com/openshift/cluster-version-operator/pkg/start.(*Options).run.func3.1.1()
      2025-02-07T10:24:37.754776401+05:30     /go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:248 +0x85
      2025-02-07T10:24:37.754776401+05:30 created by github.com/openshift/cluster-version-operator/pkg/start.(*Options).run.func3.1 in goroutine 234
      2025-02-07T10:24:37.754776401+05:30     /go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:246 +0x176
      2025-02-07T10:24:37.757971014+05:30 panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      2025-02-07T10:24:37.757971014+05:30     panic: runtime error: invalid memory address or nil pointer dereference
      2025-02-07T10:24:37.757971014+05:30 [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7c5afe] 
      ====
       

              hongkliu Hongkai Liu
              rhn-support-cchouhan Chandan Chouhan
              Jia Liu Jia Liu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: