Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32678

Panic in cluster version operator code

XMLWordPrintable

    • No
    • 5
    • OTA 256, OTA 257
    • 2
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, on rare conditions, a race condition between Go routines caused the Cluster Version Operator (CVO) to panic after the CVO started. With this release, the Go routines synchronization is improved and the issue is resolved. (link:https://issues.redhat.com/browse/OCPBUGS-32678[*OCPBUGS-32678*])
      Show
      * Previously, on rare conditions, a race condition between Go routines caused the Cluster Version Operator (CVO) to panic after the CVO started. With this release, the Go routines synchronization is improved and the issue is resolved. (link: https://issues.redhat.com/browse/OCPBUGS-32678 [* OCPBUGS-32678 *])
    • Bug Fix
    • Done

      This payload run detects a panic in CVO code. The following payloads did not see the same panic. Bug should be prioritized by CVO team accordingly.

      Relevant Job run: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-e2e-gcp-ovn-upgrade/1782008003688402944

      Panic trace as showed in this log:

      I0421 13:06:29.113325       1 availableupdates.go:61] First attempt to retrieve available updates
      I0421 13:06:29.119731       1 cvo.go:721] Finished syncing available updates "openshift-cluster-version/version" (6.46969ms)
      I0421 13:06:29.120687       1 sync_worker.go:229] Notify the sync worker: Cluster operator etcd changed Degraded from "False" to "True"
      I0421 13:06:29.120697       1 sync_worker.go:579] Cluster operator etcd changed Degraded from "False" to "True"
      E0421 13:06:29.121014       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      goroutine 185 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1bbc580?, 0x30cdc90})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x1e3efe0?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWork).calculateNextFrom(0xc002944000, 0x0)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:725 +0x58
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:584 +0x2f2
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000101800?, {0x2194c80, 0xc0026245d0}, 0x1, 0xc000118120)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x989680, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc002398c80, {0x21b41b8, 0xc0004be230}, 0x10)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:564 +0x135
      github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run.func2()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:431 +0x5d
      created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run in goroutine 118
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:429 +0x49d
      E0421 13:06:29.121188       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      goroutine 185 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1bbc580?, 0x30cdc90})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000000002?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x1e3efe0?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWork).calculateNextFrom(0xc002944000, 0x0)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:725 +0x58
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:584 +0x2f2
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000101800?, {0x2194c80, 0xc0026245d0}, 0x1, 0xc000118120)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x989680, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc002398c80, {0x21b41b8, 0xc0004be230}, 0x10)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:564 +0x135
      github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run.func2()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:431 +0x5d
      created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run in goroutine 118
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:429 +0x49d
      I0421 13:06:29.120720       1 cvo.go:738] Started syncing upgradeable "openshift-cluster-version/version"
      I0421 13:06:29.123165       1 upgradeable.go:69] Upgradeability last checked 5.274200045s ago, will not re-check until 2024-04-21T13:08:23Z
      I0421 13:06:29.123195       1 cvo.go:740] Finished syncing upgradeable "openshift-cluster-version/version" (2.469943ms)
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x195c018]
      
      goroutine 185 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000000002?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x1e3efe0?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWork).calculateNextFrom(0xc002944000, 0x0)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:725 +0x58
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:584 +0x2f2
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000101800?, {0x2194c80, 0xc0026245d0}, 0x1, 0xc000118120)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x989680, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc002398c80, {0x21b41b8, 0xc0004be230}, 0x10)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:564 +0x135
      github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run.func2()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:431 +0x5d
      created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run in goroutine 118
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:429 +0x49d
      

            dhurta@redhat.com David Hurta
            kenzhang@redhat.com Ken Zhang
            Dinesh Kumar S Dinesh Kumar S
            Olivia Brown Olivia Brown
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: