Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-3621

EUS upgrade stuck on worker pool update: error running skopeo inspect --no-tags

    XMLWordPrintable

Details

    • Sprint 227
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      EUS-to-EUS upgrade(4.10.38-4.11.13-4.12.0-rc.0), after control-plane nodes are upgraded to 4.12 successfully, unpause the worker pool to get worker nodes updated. But worker nodes failed to be updated with degraded worker pool:
      ```
      # ./oc get node
      NAME                                                   STATUS                     ROLES    AGE     VERSION
      jliu410-6hmkz-master-0.c.openshift-qe.internal         Ready                      master   4h40m   v1.25.2+f33d98e
      jliu410-6hmkz-master-1.c.openshift-qe.internal         Ready                      master   4h40m   v1.25.2+f33d98e
      jliu410-6hmkz-master-2.c.openshift-qe.internal         Ready                      master   4h40m   v1.25.2+f33d98e
      jliu410-6hmkz-worker-a-xdwvv.c.openshift-qe.internal   Ready,SchedulingDisabled   worker   4h31m   v1.23.12+6b34f32
      jliu410-6hmkz-worker-b-9hnb8.c.openshift-qe.internal   Ready                      worker   4h31m   v1.23.12+6b34f32
      jliu410-6hmkz-worker-c-bdv4f.c.openshift-qe.internal   Ready                      worker   4h31m   v1.23.12+6b34f32
      ...
      # ./oc get co machine-config
      machine-config   4.12.0-rc.0   True        False         True       3h41m   Failed to resync 4.12.0-rc.0 because: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool worker is not ready, retrying. Status: (pool degraded: true total: 3, ready 0, updated: 0, unavailable: 1)]
      ...
      # ./oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-b81233204496767f2fe32fbb6cb088e1   True      False      False      3              3                   3                     0                      4h10m
      worker   rendered-worker-a2caae543a144d94c17a27e56038d4c4   False     True       True       3              0                   0                     1                      4h10m
      ...
      # ./oc describe mcp worker
      Message:                   Reason:                    Status:                True    Type:                  Degraded    Last Transition Time:  2022-11-14T07:19:42Z    Message:               Node jliu410-6hmkz-worker-a-xdwvv.c.openshift-qe.internal is reporting: "Error checking type of update image: error running skopeo inspect --no-tags --retry-times 5 --authfile /var/lib/kubelet/config.json docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c01b0ae9870dbee5609c52b4d649334ce6854fff1237f1521929d151f6876daa: exit status 1\ntime=\"2022-11-14T07:42:47Z\" level=fatal msg=\"unknown flag: --no-tags\"\n"    Reason:                1 nodes are reporting degraded status on sync    Status:                True    Type:                  NodeDegraded
      ...
      # ./oc logs machine-config-daemon-mg2zn
      E1114 08:11:27.115577  192836 writer.go:200] Marking Degraded due to: Error checking type of update image: error running skopeo inspect --no-tags --retry-times 5 --authfile /var/lib/kubelet/config.json docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c01b0ae9870dbee5609c52b4d649334ce6854fff1237f1521929d151f6876daa: exit status 1
      time="2022-11-14T08:11:25Z" level=fatal msg="unknown flag: --no-tags"
      ```

      Version-Release number of selected component (if applicable):

      4.12.0-rc.0

      How reproducible:

       

      Steps to Reproduce:

      1. EUS upgrade with path 4.10.38-> 4.11.13-> 4.12.0-rc.0 with paused worker pool 
      2. After master pool upgrade succeed, unpause worker pool 
      3.
      

      Actual results:

      Worker pool upgrade failed

      Expected results:

      Worker pool upgrade succeed

      Additional info:

       

      Attachments

        Issue Links

          Activity

            People

              rhn-engineering-skumari Sinny Kumari
              rhn-support-jiajliu Jia Liu
              Rio Liu Rio Liu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: