-
Bug
-
Resolution: Done-Errata
-
Minor
-
4.13.z, 4.12.z, 4.14.z, 4.15.0
-
Moderate
-
No
-
False
-
This is a clone of issue OCPBUGS-20085. The following is the description of the original issue:
—
Description of problem:
During the destroy cluster operation, unexpected results from the IBM Cloud API calls for Disks can result in panics when response data (or responses) are missing, resulting in unexpected failures during destroy.
Version-Release number of selected component (if applicable):
4.15
How reproducible:
Unknown, dependent on IBM Cloud API responses
Steps to Reproduce:
1. Successfully create IPI cluster on IBM Cloud 2. Attempt to cleanup (destroy) the cluster
Actual results:
Golang panic attempting to parse a HTTP response that is missing or lacking data. level=info msg=Deleted instance "ci-op-97fkzvv2-e6ed7-5n5zg-master-0" E0918 18:03:44.787843 33 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 228 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic({0x6a3d760?, 0x274b5790}) /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xfffffffe?}) /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75 panic({0x6a3d760, 0x274b5790}) /usr/lib/golang/src/runtime/panic.go:884 +0x213 github.com/openshift/installer/pkg/destroy/ibmcloud.(*ClusterUninstaller).waitForDiskDeletion.func1() /go/src/github.com/openshift/installer/pkg/destroy/ibmcloud/disk.go:84 +0x12a github.com/openshift/installer/pkg/destroy/ibmcloud.(*ClusterUninstaller).Retry(0xc000791ce0, 0xc000573700) /go/src/github.com/openshift/installer/pkg/destroy/ibmcloud/ibmcloud.go:99 +0x73 github.com/openshift/installer/pkg/destroy/ibmcloud.(*ClusterUninstaller).waitForDiskDeletion(0xc000791ce0, {{0xc00160c060, 0x29}, {0xc00160c090, 0x28}, {0xc0016141f4, 0x9}, {0x82b9f0d, 0x4}, {0xc00160c060, ...}}) /go/src/github.com/openshift/installer/pkg/destroy/ibmcloud/disk.go:78 +0x14f github.com/openshift/installer/pkg/destroy/ibmcloud.(*ClusterUninstaller).destroyDisks(0xc000791ce0) /go/src/github.com/openshift/installer/pkg/destroy/ibmcloud/disk.go:118 +0x485 github.com/openshift/installer/pkg/destroy/ibmcloud.(*ClusterUninstaller).executeStageFunction.func1() /go/src/github.com/openshift/installer/pkg/destroy/ibmcloud/ibmcloud.go:201 +0x3f k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x7f7801e503c8, 0x18}) /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x1b k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x227a2f78?, 0xc00013c000?}, 0xc000a9b690?) /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:154 +0x57 k8s.io/apimachinery/pkg/util/wait.poll({0x227a2f78, 0xc00013c000}, 0xd0?, 0x146fea5?, 0x7f7801e503c8?) /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:245 +0x38 k8s.io/apimachinery/pkg/util/wait.PollImmediateInfiniteWithContext({0x227a2f78, 0xc00013c000}, 0x4136e7?, 0x28?) /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:229 +0x49 k8s.io/apimachinery/pkg/util/wait.PollImmediateInfinite(0x100000000000000?, 0x806f00?) /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:214 +0x46 github.com/openshift/installer/pkg/destroy/ibmcloud.(*ClusterUninstaller).executeStageFunction(0xc000791ce0, {{0x82bb9a3?, 0xc000a9b7d0?}, 0xc000111de0?}, 0x840366?, 0xc00054e900?) /go/src/github.com/openshift/installer/pkg/destroy/ibmcloud/ibmcloud.go:198 +0x108 created by github.com/openshift/installer/pkg/destroy/ibmcloud.(*ClusterUninstaller).destroyCluster /go/src/github.com/openshift/installer/pkg/destroy/ibmcloud/ibmcloud.go:172 +0xa87 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference
Expected results:
Destroy IBM Cloud Disks during cluster destroy, or provide a useful error message to follow up on.
Additional info:
The ability to reproduce is relatively low, as it requires the IBM Cloud API's to return specific data (or lack there of), which is currently unknown why the HTTP respoonse and/or data is missing. IBM Cloud already has a PR to attempt to mitigate this issue, like done with other destroy resource calls. Potentially followup for additional resources as necessary. https://github.com/openshift/installer/pull/7515
- blocks
-
OCPBUGS-29125 [IBMCloud] Unhandled response during destroy disks
- New
- clones
-
OCPBUGS-20085 [IBMCloud] Unhandled response during destroy disks
- Closed
- is blocked by
-
OCPBUGS-29123 [IBMCloud] Unhandled response during destroy disks
- Closed
- links to
-
RHBA-2024:5444 OpenShift Container Platform 4.13.z bug fix update