Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-62941

External oidc env upgrade is stuck at "748 of 954 done (78% complete), waiting on authentication over 30 minutes which is longer than expected"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.21
    • apiserver-auth
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Critical
    • None
    • None
    • None
    • Approved
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      External oidc env upgrade is stuck at "748 of 954 done (78% complete), waiting on authentication over 30 minutes which is longer than expected"

      Version-Release number of selected component (if applicable):

      Upgrading from external oidc env of version 4.21.0-0.nightly-2025-10-07-171402 to 4.21.0-0.nightly-2025-10-09-210657

      How reproducible:

      Tried twice. Both hit totally same error.

      Steps to Reproduce:

      1. Launch a 4.21.0-0.nightly-2025-10-07-171402 cluster with default featureset.
      Configure Microsoft Entra ID or the untrusted self-signed Keycloak external oidc provider. Wait for rollout.
      Before cluster upgrade, test oc login and console login, both can succeed with the external oidc provider.
      
      2. Upgrade the cluster to 4.21.0-0.nightly-2025-10-09-210657

      Actual results:

      2. After 2 hours, the upgrade is still stuck:
      $ oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.21.0-0.nightly-2025-10-07-171402   True        True          3h28m   Working towards 4.21.0-0.nightly-2025-10-09-210657: 748 of 954 done (78% complete), waiting on authentication over 30 minutes which is longer than expected

      Expected results:

      2. Upgrade should succeed.

      Additional info:

      $ oc get co # oc get co seems to not show authentication no abnormal error
      NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      authentication                             4.21.0-0.nightly-2025-10-09-210657   True        False         False      3h8m    
      baremetal                                  4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      cloud-controller-manager                   4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      cloud-credential                           4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      cluster-autoscaler                         4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      config-operator                            4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      console                                    4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      control-plane-machine-set                  4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      csi-snapshot-controller                    4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      dns                                        4.21.0-0.nightly-2025-10-07-171402   True        False         False      9h      
      etcd                                       4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      image-registry                             4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      ingress                                    4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      insights                                   4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      kube-apiserver                             4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      kube-controller-manager                    4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      kube-scheduler                             4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      kube-storage-version-migrator              4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      machine-api                                4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      machine-approver                           4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      machine-config                             4.21.0-0.nightly-2025-10-07-171402   True        False         False      9h      
      marketplace                                4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      monitoring                                 4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      network                                    4.21.0-0.nightly-2025-10-07-171402   True        False         False      9h      
      node-tuning                                4.21.0-0.nightly-2025-10-09-210657   True        False         False      3h6m    
      olm                                        4.21.0-0.nightly-2025-10-07-171402   True        False         False      9h      
      openshift-apiserver                        4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      openshift-controller-manager               4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      openshift-samples                          4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      operator-lifecycle-manager                 4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      operator-lifecycle-manager-catalog         4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      operator-lifecycle-manager-packageserver   4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      service-ca                                 4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h      
      storage                                    4.21.0-0.nightly-2025-10-09-210657   True        False         False      9h
      
      $ oc get clusterversion version -o yaml
      ...
        - lastTransitionTime: "2025-10-10T09:03:21Z"
          message: waiting on authentication over 30 minutes which is longer than expected
          reason: SlowClusterOperator
          status: Unknown
          type: Failing
        - lastTransitionTime: "2025-10-10T08:11:48Z"
          message: 'Working towards 4.21.0-0.nightly-2025-10-09-210657: 748 of 954 done
            (78% complete), waiting on authentication over 30 minutes which is longer than
            expected'
          reason: ClusterOperatorUpdating
          status: "True"
          type: Progressing
      ...
      

              rh-ee-bpalmer Bryce Palmer
              xxia-1 Xingxing Xia
              None
              None
              Xingxing Xia Xingxing Xia
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: