Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-42932

4.15-4.18 upgrade stuck on authentication operator during stage of 4.17-4.18 update

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.18
    • apiserver-auth

      Description of problem:

          Failed ci jobs:
      https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.18-multi-nightly-4.18-cpou-upgrade-from-4.15-aws-ipi-mini-perm-arm-f14/1842004955238502400
      
      https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.18-arm64-nightly-4.18-cpou-upgrade-from-4.15-azure-ipi-fullyprivate-proxy-f14/1841942041722884096
      
      The 4.15-4.18 upgrade failed at stage of 4.17 to 4.18 update while authentication operator degraded and unavailable due to APIServerDeployment_PreconditionNotFulfilled
      
      $ omc get clusterversion
      NAME      VERSION                                    AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.17.0-0.nightly-arm64-2024-10-03-172957   True        True          1h44m   Unable to apply 4.18.0-0.nightly-arm64-2024-10-03-125849: the cluster operator authentication is not available
      
      $ omc get co authentication
      NAME             VERSION                                    AVAILABLE   PROGRESSING   DEGRADED   SINCE
      authentication   4.18.0-0.nightly-arm64-2024-10-03-125849   False       False         True       8h
      
      $ omc get co authentication -ojson|jq .status.conditions[]
      {
        "lastTransitionTime": "2024-10-04T04:22:39Z",
        "message": "APIServerDeploymentDegraded: waiting for .status.latestAvailableRevision to be available\nAPIServerDeploymentDegraded: ",
        "reason": "APIServerDeployment_PreconditionNotFulfilled",
        "status": "True",
        "type": "Degraded"
      }
      {
        "lastTransitionTime": "2024-10-04T03:54:13Z",
        "message": "AuthenticatorCertKeyProgressing: All is well",
        "reason": "AsExpected",
        "status": "False",
        "type": "Progressing"
      }
      {
        "lastTransitionTime": "2024-10-04T03:52:34Z",
        "reason": "APIServerDeployment_PreconditionNotFulfilled",
        "status": "False",
        "type": "Available"
      }
      {
        "lastTransitionTime": "2024-10-03T21:32:31Z",
        "message": "All is well",
        "reason": "AsExpected",
        "status": "True",
        "type": "Upgradeable"
      }
      {
        "lastTransitionTime": "2024-10-04T00:04:57Z",
        "reason": "NoData",
        "status": "Unknown",
        "type": "EvaluationConditionsDetected"
      }

      Version-Release number of selected component (if applicable):

       4.18.0-0.nightly-arm64-2024-10-03-125849
       4.18.0-0.nightly-multi-2024-10-03-193054
      
      

      How reproducible:

          always

      Steps to Reproduce:

          1. upgrade from 4.15 to 4.16, and then to 4.17, and then to 4.18
          2.
          3.
          

      Actual results:

          upgrade stuck on authentication operator

      Expected results:

          upgrade succeed

      Additional info:

          The issue is found in a control plane only update jobs(with paused worker pool), but it's not cpou specified because it can be reproduced in a normal chain upgrade from 4.15 to 4.18 upgrade. 

            deads@redhat.com David Eads
            rhn-support-jiajliu Jia Liu
            Xingxing Xia Xingxing Xia
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: