Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32678

Panic in cluster version operator code

XMLWordPrintable

    • No
    • Approved
    • False
    • Hide

      None

      Show
      None

      This payload run detects a panic in CVO code. The following payloads did not see the same panic. Bug should be prioritized by CVO team accordingly.

      Relevant Job run: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-e2e-gcp-ovn-upgrade/1782008003688402944

      Panic trace as showed in this log:

      I0421 13:06:29.113325       1 availableupdates.go:61] First attempt to retrieve available updates
      I0421 13:06:29.119731       1 cvo.go:721] Finished syncing available updates "openshift-cluster-version/version" (6.46969ms)
      I0421 13:06:29.120687       1 sync_worker.go:229] Notify the sync worker: Cluster operator etcd changed Degraded from "False" to "True"
      I0421 13:06:29.120697       1 sync_worker.go:579] Cluster operator etcd changed Degraded from "False" to "True"
      E0421 13:06:29.121014       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      goroutine 185 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1bbc580?, 0x30cdc90})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x1e3efe0?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWork).calculateNextFrom(0xc002944000, 0x0)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:725 +0x58
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:584 +0x2f2
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000101800?, {0x2194c80, 0xc0026245d0}, 0x1, 0xc000118120)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x989680, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc002398c80, {0x21b41b8, 0xc0004be230}, 0x10)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:564 +0x135
      github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run.func2()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:431 +0x5d
      created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run in goroutine 118
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:429 +0x49d
      E0421 13:06:29.121188       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      goroutine 185 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1bbc580?, 0x30cdc90})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000000002?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x1e3efe0?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWork).calculateNextFrom(0xc002944000, 0x0)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:725 +0x58
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:584 +0x2f2
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000101800?, {0x2194c80, 0xc0026245d0}, 0x1, 0xc000118120)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x989680, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc002398c80, {0x21b41b8, 0xc0004be230}, 0x10)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:564 +0x135
      github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run.func2()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:431 +0x5d
      created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run in goroutine 118
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:429 +0x49d
      I0421 13:06:29.120720       1 cvo.go:738] Started syncing upgradeable "openshift-cluster-version/version"
      I0421 13:06:29.123165       1 upgradeable.go:69] Upgradeability last checked 5.274200045s ago, will not re-check until 2024-04-21T13:08:23Z
      I0421 13:06:29.123195       1 cvo.go:740] Finished syncing upgradeable "openshift-cluster-version/version" (2.469943ms)
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x195c018]
      
      goroutine 185 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000000002?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x1e3efe0?})
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd
      panic({0x1bbc580?, 0x30cdc90?})
      	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWork).calculateNextFrom(0xc002944000, 0x0)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:725 +0x58
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:584 +0x2f2
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000101800?, {0x2194c80, 0xc0026245d0}, 0x1, 0xc000118120)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x989680, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc002398c80, {0x21b41b8, 0xc0004be230}, 0x10)
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:564 +0x135
      github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run.func2()
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:431 +0x5d
      created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run in goroutine 118
      	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:429 +0x49d
      

            trking W. Trevor King
            kenzhang@redhat.com Ken Zhang
            Jia Liu Jia Liu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: