Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-6697

Uninstall fails with Observed a panic: runtime.boundsError

XMLWordPrintable

    • Moderate
    • False
    • Hide

      None

      Show
      None

      +++ This bug was initially created as a clone of
      Bug #2102632
      +++

      Version:
      4.11.0-0.nightly-2022-06-28-160049

      $ ./openshift-install version
      ./openshift-install 4.11.0-0.nightly-2022-06-28-160049
      built from commit 6daed68b9863a9b2ecebdf8a4056800aa5c60ad3
      release image registry.ci.openshift.org/ocp/release@sha256:b79b1be6aa4f9f62c691c043e0911856cf1c11bb81c8ef94057752c6e5a8478a
      release architecture amd64

      Platform:
      GCP

      IPI (automated install with `openshift-install`.

      What happened?
      During uninstall, the cluster uninstall I received:

      E0630 13:17:58.830361 271713 runtime.go:78] Observed a panic: runtime.boundsError{x:22, y:21, signed:true, code:0x1} (runtime error: slice bounds out of range [:22] with length 21)
      goroutine 1 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x41d43c0?, 0xc0010637e8})
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x86
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x18?})
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
      panic({0x41d43c0, 0xc0010637e8})
      /usr/lib/golang/src/runtime/panic.go:838 +0x207
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).formatClusterIDForStorage(...)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:25
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).storageIDFilter(...)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:29
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).storageLabelOrClusterIDFilter(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:39 +0x1fe
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).listDisks(0xc0015dc900?)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:43 +0x1e
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).destroyDisks(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:116 +0x36
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).destroyCluster(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/gcp.go:174 +0x78e
      k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x18, 0xc000700000})
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220 +0x1b
      k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x19f06638?, 0xc0000721c0?}, 0xc00047d888?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233 +0x57
      k8s.io/apimachinery/pkg/util/wait.poll({0x19f06638, 0xc0000721c0}, 0xc8?, 0x1108485?, 0x10?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:580 +0x38
      k8s.io/apimachinery/pkg/util/wait.PollImmediateInfiniteWithContext({0x19f06638, 0xc0000721c0}, 0x40d687?, 0x10?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:566 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateInfinite(0x19f06670?, 0xc00008b8c0?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:555 +0x46
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).Run(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/gcp.go:130 +0x519
      main.runDestroyCmd({0x7fffe6a88d87, 0x9}, 0x0)
      /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:67 +0x92
      main.newDestroyClusterCmd.func1(0xc000536780?, {0xc000906100?, 0x2?, 0x2?})
      /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:53 +0x7f
      github.com/spf13/cobra.(*Command).execute(0xc000536780, {0xc0009060c0, 0x2, 0x2})
      /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:860 +0x663
      github.com/spf13/cobra.(*Command).ExecuteC(0xc00098db80)
      /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:974 +0x3b4
      github.com/spf13/cobra.(*Command).Execute(...)
      /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:902
      main.installerMain()
      /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:60 +0x29e
      main.main()
      /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:38 +0xff
      panic: runtime error: slice bounds out of range [:22] with length 21 [recovered]
      panic: runtime error: slice bounds out of range [:22] with length 21

      goroutine 1 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x18?})
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
      panic({0x41d43c0, 0xc0010637e8})
      /usr/lib/golang/src/runtime/panic.go:838 +0x207
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).formatClusterIDForStorage(...)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:25
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).storageIDFilter(...)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:29
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).storageLabelOrClusterIDFilter(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:39 +0x1fe
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).listDisks(0xc0015dc900?)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:43 +0x1e
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).destroyDisks(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/disk.go:116 +0x36
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).destroyCluster(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/gcp.go:174 +0x78e
      k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x18, 0xc000700000})
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220 +0x1b
      k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x19f06638?, 0xc0000721c0?}, 0xc00047d888?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233 +0x57
      k8s.io/apimachinery/pkg/util/wait.poll({0x19f06638, 0xc0000721c0}, 0xc8?, 0x1108485?, 0x10?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:580 +0x38
      k8s.io/apimachinery/pkg/util/wait.PollImmediateInfiniteWithContext({0x19f06638, 0xc0000721c0}, 0x40d687?, 0x10?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:566 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateInfinite(0x19f06670?, 0xc00008b8c0?)
      /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:555 +0x46
      github.com/openshift/installer/pkg/destroy/gcp.(*ClusterUninstaller).Run(0xc000f22540)
      /go/src/github.com/openshift/installer/pkg/destroy/gcp/gcp.go:130 +0x519
      main.runDestroyCmd({0x7fffe6a88d87, 0x9}, 0x0)
      /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:67 +0x92
      main.newDestroyClusterCmd.func1(0xc000536780?, {0xc000906100?, 0x2?, 0x2?})
      /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:53 +0x7f
      github.com/spf13/cobra.(*Command).execute(0xc000536780, {0xc0009060c0, 0x2, 0x2})
      /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:860 +0x663
      github.com/spf13/cobra.(*Command).ExecuteC(0xc00098db80)
      /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:974 +0x3b4
      github.com/spf13/cobra.(*Command).Execute(...)
      /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:902
      main.installerMain()
      /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:60 +0x29e
      main.main()
      /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:38 +0xff

      Anything else we need to know?

      Uninstall with openshift-install binary from OCP 4.10.16 worked fine.

      — Additional comment from
      jmencak@redhat.com
      on 2022-06-30 12:15:49 UTC —

      Created attachment 1893636 [details]Install/uninstall directory tar ball.

      Adding install/uninstall directory tar ball.

      — Additional comment from
      padillon@redhat.com
      on 2022-07-01 14:11:22 UTC —

      Can we get an install config for the failing destroy?

      — Additional comment from
      padillon@redhat.com
      on 2022-07-01 14:13:30 UTC —

      Sorry. I see the install config is in the attachment. I thought that was only the destroy log.

      — Additional comment from
      padillon@redhat.com
      on 2022-07-01 14:28:10 UTC —

      Marking this as blocker+. It looks like
      https://github.com/openshift/installer/pull/5976
      introduced a regression when destroying disks. We should have a PR to fix up today.

      — Additional comment from
      eparis@redhat.com
      on 2022-07-01 15:00:11 UTC —

      This bug sets blocker+ without setting a Target Release. This is an invalid state as it is impossible to determine what is being blocked. Please be sure to set Priority, Severity, and Target Release before you attempt to set blocker+

      — Additional comment from
      padillon@redhat.com
      on 2022-07-01 17:56:37 UTC —

      For QE: This error would occur after installing and provisioning PV.

      — Additional comment from
      aos-team-art-private@redhat.com
      on 2022-07-05 04:27:36 UTC —

      Elliott changed bug status from MODIFIED to ON_QA.
      This bug is expected to ship in the next 4.11 release.

      — Additional comment from
      jmencak@redhat.com
      on 2022-07-07 08:38:59 UTC —

      I still see the same issue with the latest nightly 4.11.0-0.nightly-2022-07-06-145812. Is the fix included there?

      — Additional comment from
      padillon@redhat.com
      on 2022-07-07 12:51:25 UTC —

      We didn't cherry-pick this fix into 4.11 so it is not in the nightlies. You should be able to check it against a master build. We will cherry-pick to 4.11 now.

      $ oc adm release extract --tools registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-07-06-145812
      $ tar -xvf openshift-install-linux-4.11.0-0.nightly-2022-07-06-145812.tar.gz
      README.md
      openshift-install
      $ ./openshift-install version
      ./openshift-install 4.11.0-0.nightly-2022-07-06-145812
      built from commit b2e7be726e400022e71ef3b8bd01a2093e53bc5a
      release image registry.ci.openshift.org/ocp/release@sha256:616c5fefa87d116dd2440c75d9832c462078d635ed155c8d6cd486dd09540184
      release architecture amd64

      $ git show b2e7be726e400022e71ef3b8bd01a2093e53bc5a
      commit b2e7be726e400022e71ef3b8bd01a2093e53bc5a (upstream/release-4.11)
      Merge: 6daed68b9 2426260d5
      Author: openshift-ci[bot] <75433959+openshift-ci[bot]@users.noreply.github.com>
      Date: Thu Jun 30 22:11:44 2022 +0000

      Merge pull request #6060 from mike-nguyen/dnm_411_test
      Bug 2093126
      : bump RHCOS 4.11 boot image metadata
       

            rh-ee-bbarbach Brent Barbachem
            rh-ee-bbarbach Brent Barbachem
            Jianli Wei Jianli Wei
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: