-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.20
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
The oc adm upgrade status contains a table listing nodes:
Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ci-op-pw3ghqzh-bb5c4-pdhfc-master-0 Outdated Pending 4.20.0-0.nightly-2025-08-12-153542 ? ci-op-pw3ghqzh-bb5c4-pdhfc-master-1 Outdated Pending 4.20.0-0.nightly-2025-08-12-153542 ? ci-op-pw3ghqzh-bb5c4-pdhfc-master-2 Outdated Pending 4.20.0-0.nightly-2025-08-12-153542 ?
The table assumes the MESSAGE value is fairly short and displays it as the last item in the table. The message can be fairly long and can even contain linebreaks:
Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ci-op-pw3ghqzh-bb5c4-pdhfc-master-1 Degraded Updating 4.20.0-0.nightly-2025-08-12-153542 ? failed to run command nice (6 tries): [timed out waiting for the condition, running nice -- ionice -c 3 podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0 failed: time="2025-08-19T21:29:52Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:59905->168.63.129.16:53: i/o timeout" time="2025-08-19T21:30:03Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:39540->168.63.129.16:53: i/o timeout" time="2025-08-19T21:30:14Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:40554->168.63.129.16:53: i/o timeout" Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:49035->168.63.129.16:53: i/o timeout : exit status 125] ci-op-pw3ghqzh-bb5c4-pdhfc-master-0 Outdated Pending 4.20.0-0.nightly-2025-08-12-153542 ? ci-op-pw3ghqzh-bb5c4-pdhfc-master-2 Completed Updated 4.20.0-0.nightly-2025-08-19-180353 -
This is ugly and confusing, we should try finding a presentation that does not look so ugly while still capturing enough interesting information for the user to be able to investigate.
Version-Release number of selected component (if applicable):
4.20 ECs and later at least
How reproducible:
Actual condition seems to be quite rare in CI but the captured resources from the CI job make it reproducible:
Steps to Reproduce:
Reproducer added in https://github.com/openshift/oc/pull/2081
Actual results:
Ugly
Expected results:
Nice
Additional info:
This case is actually detected by a the test (example):
: [sig-cli][OCPFeatureGate:UpgradeStatus] oc adm upgrade status control plane section is consistent
Which detects an unexpected line in the control plane node table like this (pruned for clarity):
{observed unexpected outputs in oc adm upgrade status control plane section
===== 2025-08-19T21:16:00Z
= Control Plane =
...
Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ci-op-pw3ghqzh-bb5c4-pdhfc-master-1 Degraded Updating 4.20.0-0.nightly-2025-08-12-153542 ? failed to run command nice (6 tries): [timed out waiting for the condition, running nice -- ionice -c 3 podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0 failed: time="2025-08-19T21:15:11Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:57195->168.63.129.16:53: i/o timeout"
time="2025-08-19T21:15:22Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:37662->168.63.129.16:53: i/o timeout"
time="2025-08-19T21:15:33Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:48884->168.63.129.16:53: i/o timeout"
Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:53491->168.63.129.16:53: i/o timeout
: exit status 125]
ci-op-pw3ghqzh-bb5c4-pdhfc-master-0 Outdated Pending 4.20.0-0.nightly-2025-08-12-153542 ?
ci-op-pw3ghqzh-bb5c4-pdhfc-master-2 Completed Updated 4.20.0-0.nightly-2025-08-19-180353 -
= Worker Upgrade =
...
= Update Health =
...
=> Bad line in nodes: : exit status 125]