Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60768

oc adm upgrade status poorly displays long / multiline node messages

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.20
    • oc / update
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      The oc adm upgrade status contains a table listing nodes:

      Control Plane Nodes
      NAME                                  ASSESSMENT   PHASE     VERSION                              EST   MESSAGE
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-0   Outdated     Pending   4.20.0-0.nightly-2025-08-12-153542   ?     
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-1   Outdated     Pending   4.20.0-0.nightly-2025-08-12-153542   ?     
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-2   Outdated     Pending   4.20.0-0.nightly-2025-08-12-153542   ?     
      

      The table assumes the MESSAGE value is fairly short and displays it as the last item in the table. The message can be fairly long and can even contain linebreaks:

      Control Plane Nodes
      NAME                                  ASSESSMENT   PHASE      VERSION                              EST   MESSAGE
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-1   Degraded     Updating   4.20.0-0.nightly-2025-08-12-153542   ?     failed to run command nice (6 tries): [timed out waiting for the condition, running nice -- ionice -c 3 podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0 failed: time="2025-08-19T21:29:52Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:59905->168.63.129.16:53: i/o timeout"
      time="2025-08-19T21:30:03Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:39540->168.63.129.16:53: i/o timeout"
      time="2025-08-19T21:30:14Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:40554->168.63.129.16:53: i/o timeout"
      Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:49035->168.63.129.16:53: i/o timeout
      : exit status 125]
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-0   Outdated    Pending   4.20.0-0.nightly-2025-08-12-153542   ?   
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-2   Completed   Updated   4.20.0-0.nightly-2025-08-19-180353   -   
      

      This is ugly and confusing, we should try finding a presentation that does not look so ugly while still capturing enough interesting information for the user to be able to investigate.

      Version-Release number of selected component (if applicable):
      4.20 ECs and later at least

      How reproducible:
      Actual condition seems to be quite rare in CI but the captured resources from the CI job make it reproducible:

      Steps to Reproduce:
      Reproducer added in https://github.com/openshift/oc/pull/2081

      Actual results:
      Ugly

      Expected results:
      Nice

      Additional info:

      This case is actually detected by a the test (example):

      : [sig-cli][OCPFeatureGate:UpgradeStatus] oc adm upgrade status control plane section is consistent
      

      Which detects an unexpected line in the control plane node table like this (pruned for clarity):

      {observed unexpected outputs in oc adm upgrade status control plane section  
      ===== 2025-08-19T21:16:00Z
      = Control Plane =
      ...
      
      Control Plane Nodes
      NAME                                  ASSESSMENT   PHASE      VERSION                              EST   MESSAGE
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-1   Degraded     Updating   4.20.0-0.nightly-2025-08-12-153542   ?     failed to run command nice (6 tries): [timed out waiting for the condition, running nice -- ionice -c 3 podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0 failed: time="2025-08-19T21:15:11Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:57195->168.63.129.16:53: i/o timeout"
      time="2025-08-19T21:15:22Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:37662->168.63.129.16:53: i/o timeout"
      time="2025-08-19T21:15:33Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:48884->168.63.129.16:53: i/o timeout"
      Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56bab0cd6a692bb7021640889ca69a11d6fc6df1af210c4e24d4feef81c6d7b0: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 168.63.129.16:53: read udp 10.0.0.6:53491->168.63.129.16:53: i/o timeout
      : exit status 125]
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-0   Outdated    Pending   4.20.0-0.nightly-2025-08-12-153542   ?   
      ci-op-pw3ghqzh-bb5c4-pdhfc-master-2   Completed   Updated   4.20.0-0.nightly-2025-08-19-180353   -   
      
      = Worker Upgrade =
      ...
      
      = Update Health =
      ...
      
      => Bad line in nodes: : exit status 125]
      

              Unassigned Unassigned
              afri@afri.cz Petr Muller
              None
              None
              Dinesh Kumar S Dinesh Kumar S
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: