Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: 4.20.0
Affects Version/s: 4.18
Component/s: Installer / openshift-installer
Labels:
- triaged

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
Moderate
Regression:
None

Target Backport Versions:

4.18, 4.19
Target Version:

4.20.0
Release Blocker:
None
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Release Note Status:
Done
Release Note Type:
Bug Fix
Release Note Text:

Hide
* Before this update, if the proxy configuration for an {azure-first} cluster was deleted while installing a cluster, the program reported an unreadable error and the proxy connection timed out. With this release, when the proxy configuration for the cluster is deleted while installing a cluster, the program reports a readable error message and the issue is resolved. (link:https://issues.redhat.com/browse/OCPBUGS-45805[~~OCPBUGS-45805~~])

Show
* Before this update, if the proxy configuration for an {azure-first} cluster was deleted while installing a cluster, the program reported an unreadable error and the proxy connection timed out. With this release, when the proxy configuration for the cluster is deleted while installing a cluster, the program reports a readable error message and the issue is resolved. (link: https://issues.redhat.com/browse/OCPBUGS-45805 [ OCPBUGS-45805 ])

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Description of problem:

Run command "openshift-install destroy cluster" behind proxy, during destroying resource group, proxy is down and not recovered in time, installer still continued to delete resource group, proxy connection timeout error is shown in log, and threw out panic error finally after 1hour.

This might be corner case, but installer should show readable log instead of panic error.

$ ./openshift-install destroy cluster --dir ipi/ --log-level debug
DEBUG OpenShift Installer 4.18.0-0.nightly-2024-12-05-103644 
DEBUG Built from commit 68ca794a3e2228e0eb171fd9a62d9a648125db0c 
INFO Credentials loaded from file "/home/fedora/.azure/osServicePrincipal.json" 
DEBUG found resource group name=jima06a-hqh8b-rg from tags 
DEBUG found network resource group name= from tags 
DEBUG deleting public records                      
DEBUG removing matching private records from qe.azure.devcluster.openshift.com 
INFO deleted                                       record=api.jima06a
INFO deleted                                       record=*.apps.jima06a
DEBUG deleting resource group                      DEBUG failed to delete jima06a-hqh8b-rg: pollingTrackerBase#pollForStatus: failed to send HTTP request: StatusCode=0 -- Original Error: Get "https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1KSU1BMDZBOjJESFFIOEI6MkRSRy1FQVNUVVMiLCJqb2JMb2NhdGlvbiI6ImVhc3R1cyJ9?api-version=2018-05-01&t=638690575134777418&c=MIIHpTCCBo2gAwIBAgITOgOyui871wRtge60rAAEA7K6LzANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSU5GUkEgQ0EgMDEwHhcNMjQwOTIyMTIyOTEzWhcNMjUwMzIxMTIyOTEzWjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALthCEIs4vwfpBtc8a6akzk9FNp-luTjsKhTETKForQzaFdF5S6PDjwZJVAOcVWePFAecP08Qp1FzfiX_Cl9StTirn6FO-MjJbWSG2THMvz756N87v6UNTXfNr29-Y-iFxGXE0LRwZ39GG4hV9nUekLb8OlN6VC48-A54O0iJybH12xGD4eKLbn4ilMqeYCyiivgk_AAxCvO75VjkDUu1PztXTqXfeLvWxaeqT1RTux_k8SPeFde5JPpWGKFNih2uv3JG5KjyszNBV6b2MaWQRiNYIiPo9N_DvTiv2r7BWNvocSISt9wLymfNp0mmAvGajmSbXqs2iphrYgHuVIhOskCAwEAAaOCBJIwggSOMCcGCSsGAQQBgjcVCgQaMBgwCgYIKwYBBQUHAwEwCgYIKwYBBQUHAwIwPQYJKwYBBAGCNxUHBDAwLgYmKwYBBAGCNxUIhpDjDYTVtHiE8Ys-hZvdFs6dEoFghfmRS4WsmTQCAWQCAQcwggHaBggrBgEFBQcBAQSCAcwwggHIMGYGCCsGAQUFBzAChlpodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vcGtpaW5mcmEvQ2VydHMvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmwxLmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MFYGCCsGAQUFBzAChkpodHRwOi8vY3JsMi5hbWUuZ2JsL2FpYS9CWTJQS0lJTlRDQTAxLkFNRS5HQkxfQU1FJTIwSU5GUkElMjBDQSUyMDAxKDQpLmNydDBWBggrBgEFBQcwAoZKaHR0cDovL2NybDMuYW1lLmdibC9haWEvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmw0LmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MB0GA1UdDgQWBBTccG012vq9QsZJa-EMbFVNK7oNDzAOBgNVHQ8BAf8EBAMCBaAwggE1BgNVHR8EggEsMIIBKDCCASSgggEgoIIBHIZCaHR0cDovL2NybC5taWNyb3NvZnQuY29tL3BraWluZnJhL0NSTC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMS5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMi5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMy5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsNC5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JsMIGdBgNVHSAEgZUwgZIwDAYKKwYBBAGCN3sBATBmBgorBgEEAYI3ewICMFgwVgYIKwYBBQUHAgIwSh5IADMAMwBlADAAMQA5ADIAMQAtADQAZAA2ADQALQA0AGYAOABjAC0AYQAwADUANQAtADUAYgBkAGEAZgBmAGQANQBlADMAMwBkMAwGCisGAQQBgjd7AwEwDAYKKwYBBAGCN3sEAjAfBgNVHSMEGDAWgBTl2Ztn_PjsurvwwKidileIud8-YzAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADggEBAHkWc1KMstXAU5p0ciSSzPoB_W772N_r2ITPfnNv7OdZGLyDL8VTUXFgLP_lVJgsrUcSXJi6GSjDB9ww5vQCsXJQgzO4bCx-jKgHjqphvAtqaiiUEs6HJZYAXzjGr6BPLIlx2qov7eiI-vLqT3h9gqiUGYjLwwKJdO4PIGGzt5oIY5oWtLkqoxoad8CfpX7z8WM9YAUv1l-majml_BXWMmN0t5BbCB5MAuZPnD2rg3mtvh4FEV5pnTtTLPwpw7AAt-9DReZQOgsVkBLNSAzvuXHJzMAm03Gr-uZw7fY_-p1aqDgZ4KMTnQqVr8NvFE02_Gj_5hIaAdCp1bUhZ3ZrmZY&s=UZ0BtzcgLT_aHvONYJ0FES49Y6O4GiGcczHTPR-Ogm8YlGjGxokJXHAP3K-DkE6955-ck02ubchnpI89e-zGsLKVtn8Z4TRtO2451r_NBzTKRp2mue4o33zuV8OFrm8vVcuUWWc0oH53g4yJueNL49dMlTAnna0MyDqgxbRXsNxwuFsk2lQ1-jhFUKZ_ieVZg7Yhl9556qbn38oat0_067yg29InOjqIqr9phI4O25GUKz5mib4-EmqffgQ-S0rvVoVoERUydC0ttxDVRoCSI5esp1oUk-_xAaWkUH6u9eHYCNowm3yOjGMflyXGz5uHKBFfrH9u0k5Y4erVugectQ&h=WiXHrjkaQOrMOgvG2PjMciRcgXIn-lzXzobbO755w2I": proxyconnect tcp: dial tcp 172.191.66.148:3128: i/o timeout 
DEBUG deleting resource group                      
DEBUG failed to delete jima06a-hqh8b-rg: resources.GroupsClient#Delete: Failure sending request: StatusCode=0 -- Original Error: Delete "https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourcegroups/jima06a-hqh8b-rg?api-version=2018-05-01": proxyconnect tcp: dial tcp 172.191.66.148:3128: i/o timeout 
DEBUG deleting resource group                      
DEBUG failed to delete jima06a-hqh8b-rg: resources.GroupsClient#Delete: Failure sending request: StatusCode=0 -- Original Error: Delete "https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourcegroups/jima06a-hqh8b-rg?api-version=2018-05-01": proxyconnect tcp: dial tcp 172.191.66.148:3128: i/o timeout 
DEBUG deleting resource group                      
DEBUG failed to delete jima06a-hqh8b-rg: resources.GroupsClient#Delete: Failure sending request: StatusCode=0 -- Original Error: Delete "https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourcegroups/jima06a-hqh8b-rg?api-version=2018-05-01": proxyconnect tcp: dial tcp 172.191.66.148:3128: i/o timeout 
DEBUG deleting resource group                      
DEBUG failed to delete jima06a-hqh8b-rg: resources.GroupsClient#Delete: Failure sending request: StatusCode=0 -- Original Error: Delete "https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourcegroups/jima06a-hqh8b-rg?api-version=2018-05-01": proxyconnect tcp: dial tcp 172.191.66.148:3128: i/o timeout 
DEBUG deleting resource group                      
DEBUG failed to delete jima06a-hqh8b-rg: resources.GroupsClient#Delete: Failure sending request: StatusCode=0 -- Original Error: Delete "https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourcegroups/jima06a-hqh8b-rg?api-version=2018-05-01": proxyconnect tcp: dial tcp 172.191.66.148:3128: i/o timeout 
DEBUG deleting resource group                      
DEBUG failed to delete jima06a-hqh8b-rg: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: ClientSecretCredential: FromClientSecret(): server response error: 
DEBUG  Post "https://login.microsoftonline.com/6047c7e9-b2ad-488d-a54e-dc3f6be6a7ee/oauth2/v2.0/token": proxyconnect tcp: dial tcp 172.191.66.148:3128: i/o timeout 
E1206 05:48:29.097631 2096254 panic.go:261] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer dereference\"" stacktrace=<
    goroutine 1 [running]:
    k8s.io/apimachinery/pkg/util/runtime.logPanic({0x226ec990, 0x2731c160}, {0x6e74d80, 0x271927c0})
        /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc
    k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x226ec990, 0x2731c160}, {0x6e74d80, 0x271927c0}, {0x2731c160, 0x0, 0x43d585?})
        /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e
    k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0000061c0?})
        /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
    panic({0x6e74d80?, 0x271927c0?})
        /usr/lib/golang/src/runtime/panic.go:770 +0x132
    github.com/openshift/installer/pkg/destroy/azure.isAuthError({0x2265cf20, 0xc000c380a0})
        /go/src/github.com/openshift/installer/pkg/destroy/azure/azure.go:679 +0xdf
    github.com/openshift/installer/pkg/destroy/azure.(*ClusterUninstaller).Run.func2({0x226ecac0, 0xc000a07dc0})
        /go/src/github.com/openshift/installer/pkg/destroy/azure/azure.go:223 +0x1b3
    k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2(0xc001a8d458?, {0x226ecac0?, 0xc000a07dc0?})
        /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:87 +0x52
    k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x226ecac0, 0xc000a07dc0}, {0x226c8820, 0xc000fabf60}, 0x0, 0x0, 0xc0011a37f8)
        /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:88 +0x24d
    k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel({0x226ecac0, 0xc000a07dc0}, 0xc000a07dc0?, 0x0, 0xc001a3f7f8)
        /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:33 +0x56
    github.com/openshift/installer/pkg/destroy/azure.(*ClusterUninstaller).Run(0xc000c5c008)
        /go/src/github.com/openshift/installer/pkg/destroy/azure/azure.go:214 +0xb3b
    main.runDestroyCmd({0x7fff8e41d453, 0x4}, 0x0)
        /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:70 +0xca
    main.newDestroyCmd.newDestroyClusterCmd.func3(0xc0001ffb00?, {0xc000e4d8c0?, 0x4?, 0x887d3b3?})
        /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:55 +0x74
    github.com/spf13/cobra.(*Command).execute(0xc0009a8908, {0xc000e4d880, 0x4, 0x4})
        /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:989 +0xab1
    github.com/spf13/cobra.(*Command).ExecuteC(0xc000c77808)
        /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff
    github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1041
    main.installerMain()
        /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x3c6
    main.main()
        /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168
 >
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x5e7313f]goroutine 1 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x226ec990, 0x2731c160}, {0x6e74d80, 0x271927c0}, {0x2731c160, 0x0, 0x43d585?})
    /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0000061c0?})
    /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
panic({0x6e74d80?, 0x271927c0?})
    /usr/lib/golang/src/runtime/panic.go:770 +0x132
github.com/openshift/installer/pkg/destroy/azure.isAuthError({0x2265cf20, 0xc000c380a0})
    /go/src/github.com/openshift/installer/pkg/destroy/azure/azure.go:679 +0xdf
github.com/openshift/installer/pkg/destroy/azure.(*ClusterUninstaller).Run.func2({0x226ecac0, 0xc000a07dc0})
    /go/src/github.com/openshift/installer/pkg/destroy/azure/azure.go:223 +0x1b3
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2(0xc001a8d458?, {0x226ecac0?, 0xc000a07dc0?})
    /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:87 +0x52
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x226ecac0, 0xc000a07dc0}, {0x226c8820, 0xc000fabf60}, 0x0, 0x0, 0xc0011a37f8)
    /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:88 +0x24d
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel({0x226ecac0, 0xc000a07dc0}, 0xc000a07dc0?, 0x0, 0xc001a3f7f8)
    /go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:33 +0x56
github.com/openshift/installer/pkg/destroy/azure.(*ClusterUninstaller).Run(0xc000c5c008)
    /go/src/github.com/openshift/installer/pkg/destroy/azure/azure.go:214 +0xb3b
main.runDestroyCmd({0x7fff8e41d453, 0x4}, 0x0)
    /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:70 +0xca
main.newDestroyCmd.newDestroyClusterCmd.func3(0xc0001ffb00?, {0xc000e4d8c0?, 0x4?, 0x887d3b3?})
    /go/src/github.com/openshift/installer/cmd/openshift-install/destroy.go:55 +0x74
github.com/spf13/cobra.(*Command).execute(0xc0009a8908, {0xc000e4d880, 0x4, 0x4})
    /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:989 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0xc000c77808)
    /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
    /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1041
main.installerMain()
    /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x3c6
main.main()
    /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168

Version-Release number of selected component (if applicable):

    4.18 nightly build

How reproducible:

    Always

Steps to Reproduce:

    1. Run command "openshift-install destroy cluster" behind proxy
    2. Cut off proxy during deleting resource group
    3.

Actual results:

    Installer threw out panic error

Expected results:

    Installer should show up readable log to end user instead of panic error

Additional info:

links to

openshift/installer#9488: OCPBUGS-45805: Check for nil in case of proxy crash/deleted

Assignee:: Aditya Narayanaswamy

Reporter:: Jinyun Ma

Need Info From:: None

Contributors:: None

QA Contact:: Jinyun Ma

Doc Contact:: None

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2024/12/06 8:10 AM

Updated:: 2025/10/21 4:16 AM

Resolved:: 2025/10/21 4:16 AM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates