Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-44483

[Premerge test] OAuth APIs and stuff are not removed after OCP BYO external OIDC configuration is configured

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • premerge
    • oauth-apiserver
    • None
    • Critical
    • None
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      This is a bug found during pre-merge test of 4.18 epic AUTH-528 PRs and filed for better tracking per existing "OpenShift - Testing Before PR Merges - Left-Shift Testing" google doc workflow.

      After OCP BYO external OIDC configuration is configured, OAuth resources (oauth-openshift and oauth-apiserver pods) and associated APIs and stuff are not removed, though they do not work any more and the BYO external OIDC mechanism works well (i.e. `oc login --exec-plugin=oc-oidc ...snipped...` and console login to the external OIDC both work well)
      

      Version-Release number of selected component (if applicable):

      The cluster-bot build that is built at 2024-11-13 07:38 CST (UTC+800):
      build openshift/cluster-authentication-operator#713,openshift/cluster-kube-apiserver-operator#1760

      How reproducible:

      Always

      Steps to Reproduce:

      1. Launch a TechPreviewNoUpgrade cluster with above cluster-bot build that is built at 2024-11-13 07:38 CST (UTC+800).
      
      2. Before BYO external OIDC configuration, configure OAuth IDP and prepare some OAuth resources (users, oauthaccesstokens) for later check:
      $ oc login -u testuser-10 -p ...snipped...
      Login successful.
      ...
      
      $ oc whoami -t
      sha256~JmDE...snipped...
      
      $ oc get user
      NAME          UID                                    FULL NAME   IDENTITIES
      testuser-10   45186db0-c152-4f01-a3cb-c3a4e951f201               flexy-htpasswd-provider:testuser-10
      
      $ oc get oauthaccesstoken
      NAME                                                 USER NAME     CLIENT NAME                    CREATED   EXPIRES                         REDIRECT URI                                                                            SCOPES
      sha256~4GqfMvEQ-LuH-AbTI3CGYNETAovEdpTLTr-NN8qlmvY   testuser-10   openshift-challenging-client   3m37s     2024-11-14 02:32:03 +0000 UTC   https://...snipped.../oauth/token/implicit   user:full
      
      3. Configure BYO external OIDC:
      # "OPENSHIFT DEV" tenant
      ISSUER_URL=https://login.microsoftonline.com/...snipped...a7ee/v2.0
      CONSOLE_CLIENT_ID=...snipped...b778
      CONSOLE_CLIENT_SECRET_VALUE='...snipped...Wb04' 
      CONSOLE_CLIENT_SECRET_NAME=console-secret
      CLI_CLIENT_ID=...snipped...900e
      AUDIENCE_1=$CONSOLE_CLIENT_ID
      AUDIENCE_2=$CLI_CLIENT_ID
      
      oc create secret generic $CONSOLE_CLIENT_SECRET_NAME --from-literal=clientSecret=$CONSOLE_CLIENT_SECRET_VALUE -n openshift-config
      
      oc patch authentication.config/cluster --type=merge -p="
      spec:
        oidcProviders:
        - claimMappings:
            groups:
              claim: groups
              prefix: 'oidc-groups-test:'
            username:
              claim: email
              prefixPolicy: Prefix
              prefix:
                prefixString: 'oidc-user-test:'
          issuer:
            audiences:
            - $AUDIENCE_1
            - $AUDIENCE_2
            issuerURL: $ISSUER_URL
          name: microsoft-entra-id
          oidcClients:
          - clientID: $CONSOLE_CLIENT_ID
            clientSecret:
              name: $CONSOLE_CLIENT_SECRET_NAME
            componentName: console
            componentNamespace: openshift-console
        type: OIDC
        webhookTokenAuthenticator: null
      "
      
      Wait for KAS pods and console pods to complete rotation.
      
      4. Test `oc login --exec-plugin=oc-oidc ...snipped...` and console login to the external OIDC. Both work well.
      $ oc login --exec-plugin=oc-oidc --issuer-url=$ISSUER_URL --client-id=$CLI_CLIENT_ID --extra-scopes=email,profile --callback-port=8080
      Logged into "https://api...snipped...:6443" as "oidc-user-test:xxia@redhat.com" from an external oidc issuer.
      
      You don't have any projects. Contact your system administrator to request a project.
      
      5. OAuth does not work any more which is expected by design.
      $ oc login -u testuser-10 -p ...snipped...
      error: couldn't get https://api....snipped...:6443/.well-known/oauth-authorization-server: unexpected response status 404
      $ oc login -u kubeadmin -p ...snipped...
      error: couldn't get https://api....snipped...:6443/.well-known/oauth-authorization-server: unexpected response status 404
      
      $ oc login --token sha256~Z4NE...snipped...
      error: The token provided is invalid or expired.
      
      6. Check OAuth APIs and stuff:
      $ oc get apiservices | grep -e user -e oauth
      v1.oauth.openshift.io                         openshift-oauth-apiserver/api                                True        162m
      v1.user.openshift.io                          openshift-oauth-apiserver/api                                True        162m
      
      $ oc get po -n openshift-authentication
      NAME                               READY   STATUS    RESTARTS   AGE
      oauth-openshift-744d78dd75-7fsdl   1/1     Running   0          140m
      oauth-openshift-744d78dd75-7pwpd   1/1     Running   0          141m
      oauth-openshift-744d78dd75-b6ngk   1/1     Running   0          140m
      
      $ oc get po -n openshift-oauth-apiserver
      NAME                         READY   STATUS    RESTARTS   AGE
      apiserver-64f859977c-2mgqw   1/1     Running   0          150m
      apiserver-64f859977c-48lkb   1/1     Running   0          152m
      apiserver-64f859977c-hcqhs   1/1     Running   0          151m
      
      $ oc get user
      NAME          UID                                    FULL NAME   IDENTITIES
      testuser-10   45186db0-c152-4f01-a3cb-c3a4e951f201               flexy-htpasswd-provider:testuser-10
      
      $ oc get oauthaccesstoken
      NAME                                                 USER NAME     CLIENT NAME                    CREATED   EXPIRES                         REDIRECT URI                                                                            SCOPES
      sha256~4GqfMvEQ-LuH-AbTI3CGYNETAovEdpTLTr-NN8qlmvY   testuser-10   openshift-challenging-client   129m     2024-11-14 02:32:03 +0000 UTC   https://...snipped.../oauth/token/implicit   user:full
      
      7. Visit https://...snipped.../oauth/token/request in browser. It still shows the old kube:admin and htpasswd IDPs! Click htpasswd IDP and input user/password, it still shows "Display token". Click "Display token", it still shows below content though below oc login command does not work any more:
      ```
      Your API token is
      sha256~TW9W...snipped...
      Log in with this token
      
      oc login --token=sha256~TW9W...snipped... --server=https://api....snipped...:6443
      ...
      ```
      
      8. Check more stuffs:
      
      "integratedOAuthMetadata" still shows:
      $ oc get authentication.config/cluster -o yaml
      ...
      spec:
        oauthMetadata:
          name: ""
        oidcProviders:
        - claimMappings:
      ...
      status:
        integratedOAuthMetadata:
          name: oauth-openshift
        oidcClients:
      ...
      
      "oauth" still shows:
      $ oc get co authentication -o yaml | grep oauth
          resource: oauths
          name: oauth-openshift
          name: oauth-openshift
          name: openshift-oauth-apiserver
        - name: oauth-apiserver
        - name: oauth-openshift
      

      Actual results:

      See above step 6 ~ 8.

      Expected results:

      Given step 5 shows OAuth does not work any more, step 6 should remove original OAuth pods and OAuth APIs (apiservices) and/or resources like users/oauthaccesstokens.
      
      And step 7 should not have https://...snipped.../oauth/token/request still responsive and still show old IDPs and still return oauth tokens.
      
      And step 8 should remove all "oauth" traces in the YAML output.

      Additional info:

       

              Unassigned Unassigned
              xxia-1 Xingxing Xia
              Xingxing Xia Xingxing Xia
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: