Version:

$ openshift-install version
./openshift-install 4.11.0-0.nightly-ppc64le-2022-06-16-003709
built from commit f39a15787e72cd803e450500ce6be9ff846e5b81
release image quay.io/openshift-release-dev/ocp-release-nightly@sha256:1887990a385be4d835ccd83a0ff938bb3b9a117686364a10a6014c7f0170f180
release architecture ppc64le

Platform:

powervs
IPI

What happened?

Cluster creation failed. The first destroy timed out. The second destroy completed successfully.

The first destroy shows this error:

DEBUG Waiting for job "78c9630b-55e2-451b-a5ab-1ff094d4c054" to delete (status is "running")
DEBUG Waiting for job "78c9630b-55e2-451b-a5ab-1ff094d4c054" to delete (status is "running")
DEBUG Waiting for job "78c9630b-55e2-451b-a5ab-1ff094d4c054" to delete (status is "running")
DEBUG Waiting for job "78c9630b-55e2-451b-a5ab-1ff094d4c054" to delete (status is "running")
DEBUG Waiting for job "78c9630b-55e2-451b-a5ab-1ff094d4c054" to delete (status is "running")
DEBUG Waiting for job "78c9630b-55e2-451b-a5ab-1ff094d4c054" to delete (status is "running")
DEBUG powervs.PolledRun: Failed destroyCluster
DEBUG powervs.Run: after wait.PollImmediateInfinite, err = failed to destroy cluster: destroyCluster: timed out
FATAL Failed to destroy cluster: failed to destroy cluster: destroyCluster: timed out

The second destroy command always works.

INFO Deleted job "7f19e4d5-7dc2-40a0-b4e3-48599d293d8a"
DEBUG destroyCluster: <-wgDone
DEBUG executeStageFunction: Adding: DNS Records
DEBUG executeStageFunction: Adding: SSH Keys
DEBUG executeStageFunction: duration = 14m59.999996042s
DEBUG executeStageFunction: duration = 14m59.999998285s
DEBUG executeStageFunction: Adding: Cloud Object Storage Instances
DEBUG executeStageFunction: Executing: DNS Records
DEBUG executeStageFunction: duration = 14m59.999996356s
DEBUG Listing DNS records
DEBUG executeStageFunction: Executing: Cloud Object Storage Instances
DEBUG executeStageFunction: Executing: SSH Keys
DEBUG Listing COS instances
DEBUG Listing SSHKeys
DEBUG listSSHKeys: FOUND: rdr-kumarmn-p57x5-key
DEBUG listCOSInstances: FOUND rdr-kumarmn-p57x5-cos dd615706-7c65-4cde-8e7e-87d59caefbb5
DEBUG listDNSRecords: FOUND: da860eb4509ac0379c3f792cb05c7009, api-int.rdr-kumarmn.scnl-ibm.com
DEBUG listDNSRecords: FOUND: 68e20d7ec80e01bba689b3f14dbb56be, api.rdr-kumarmn.scnl-ibm.com
DEBUG listDNSRecords: PerPage = 20, Page = 1, Count = 20
DEBUG listDNSRecords: moreData = true
DEBUG Deleting sshKey "rdr-kumarmn-p57x5-key"
DEBUG listDNSRecords: PerPage = 20, Page = 2, Count = 20
DEBUG listDNSRecords: moreData = true
DEBUG listDNSRecords: PerPage = 20, Page = 3, Count = 4
DEBUG listDNSRecords: moreData = false
DEBUG Deleting DNS record "api.rdr-kumarmn.scnl-ibm.com"
INFO Deleted sshKey "rdr-kumarmn-p57x5-key"
DEBUG Deleting DNS record "api-int.rdr-kumarmn.scnl-ibm.com"
DEBUG Deleting COS instance "rdr-kumarmn-p57x5-cos"
INFO Deleted DNS record "api.rdr-kumarmn.scnl-ibm.com"
INFO Deleted DNS record "api-int.rdr-kumarmn.scnl-ibm.com"
INFO Deleted COS instance "rdr-kumarmn-p57x5-cos"
DEBUG destroyCluster: <-wgDone
DEBUG powervs.Run: after wait.PollImmediateInfinite, err = <nil>
DEBUG Purging asset "Metadata" from disk
DEBUG Purging asset "Master Ignition Customization Check" from disk
DEBUG Purging asset "Worker Ignition Customization Check" from disk
DEBUG Purging asset "Terraform Variables" from disk
DEBUG Purging asset "Kubeconfig Admin Client" from disk
DEBUG Purging asset "Kubeadmin Password" from disk
DEBUG Purging asset "Certificate (journal-gatewayd)" from disk
DEBUG Purging asset "Cluster" from disk
INFO Time elapsed: 2m34s

What did you expect to happen?

Expected destroy to work the first time, and not have to run it a second time.

How to reproduce it (as minimally and precisely as possible)?

$ ./openshift-install destroy cluster --log-level=debug --dir=test
$ ./openshift-install destroy cluster --log-level=debug --dir=test