Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18674

18 upgrade failures due to console cluster operator while upgrading 4.12.29 SNOs to 4.13.9

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      Sometimes, the Console Operator status can get stuck in a failed state when the Operator is removed.
      To work around this issue, patch the Console controller to reset all conditions to the default state when the Console Operator is removed.
      For example, log in to the cluster and run the following command:

      [source,terminal]
      ----
      $ oc patch console.operator.openshift.io/cluster --type='merge' -p='{"status":{"conditions":null}}'
      ----
      Show
      Sometimes, the Console Operator status can get stuck in a failed state when the Operator is removed. To work around this issue, patch the Console controller to reset all conditions to the default state when the Console Operator is removed. For example, log in to the cluster and run the following command: [source,terminal] ---- $ oc patch console.operator.openshift.io/cluster --type='merge' -p='{"status":{"conditions":null}}' ----
    • Bug Fix
    • Done

      Description of problem:

      While running an upgrade test upgrading 3555 4.12.29 SNOs to 4.13.9, 18 SNOs ended up in partial upgrade status.  They all failed on console operator related issues.  This appears to be the same bug as this https://issues.redhat.com/browse/OCPBUGS-1043

# cat partial-upgrades | xargs -I % sh -c 'echo -n "% "; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get clusterversion --no-headers' 
vm00025 version   4.12.29   True   True   40h   Unable to apply 4.13.9: the cluster operator console is not available
vm00512 version   4.12.29   True   True   41h   Unable to apply 4.13.9: the cluster operator console is not available
vm00674 version   4.12.29   True   True   41h   Unable to apply 4.13.9: the cluster operator console is not available
vm01093 version   4.12.29   True   True   38h   Unable to apply 4.13.9: the cluster operator console is not available
vm01173 version   4.12.29   True   True   38h   Unable to apply 4.13.9: the cluster operator console is not available
vm01201 version   4.12.29   True   True   38h   Unable to apply 4.13.9: the cluster operator console is not available
vm01353 version   4.12.29   True   True   38h   Unable to apply 4.13.9: wait has exceeded 40 minutes for these operators: console
vm01396 version   4.12.29   True   True   38h   Unable to apply 4.13.9: the cluster operator console is not available
vm01513 version   4.12.29   True   True   38h   Unable to apply 4.13.9: the cluster operator console is not available
vm01740 version   4.12.29   True   True   38h   Unable to apply 4.13.9: the cluster operator console is not available
vm01921 version   4.12.29   True   True   38h   Unable to apply 4.13.9: the cluster operator console is not available
vm02290 version   4.12.29   True   True   36h   Unable to apply 4.13.9: the cluster operator console is not available
vm02606 version   4.12.29   True   True   36h   Unable to apply 4.13.9: the cluster operator console is not available
vm02642 version   4.12.29   True   True   36h   Unable to apply 4.13.9: the cluster operator console is not available
vm02784 version   4.12.29   False   True   36h   Unable to apply 4.13.9: the cluster operator console is not available
vm02849 version   4.12.29   True   True   36h   Unable to apply 4.13.9: wait has exceeded 40 minutes for these operators: console
vm03354 version   4.12.29   True   True   34h   Unable to apply 4.13.9: wait has exceeded 40 minutes for these operators: console
vm03505 version   4.12.29   True   True   34h   Unable to apply 4.13.9: the cluster operator console is not available

      Version-Release number of selected component (if applicable):

      Hub cluster is 4.13.10
SNOs were originally deployed as 4.12.29, then upgraded to 4.13.9
ACM version is 2.9.0-DOWNSTREAM-2023-09-01-02-58-15

      How reproducible:

       

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

       

      Expected results:

       

      Additional info:

       

        1. must-gather-vm00025.tar.gz
          76.93 MB
          Alex Krzos
        2. must-gather-vm00512.tar.gz
          80.63 MB
          Alex Krzos
        3. must-gather-vm01353.tar.gz
          72.61 MB
          Alex Krzos
        4. must-gather-vm01396.tar.gz
          68.10 MB
          Alex Krzos
        5. must-gather-vm02849.tar.gz
          72.98 MB
          Alex Krzos

            rh-ee-jonjacks Jon Jackson
            akrzos@redhat.com Alex Krzos
            Yanping Zhang Yanping Zhang
            Aidan Reilly Aidan Reilly
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: