Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-35335

Failed to pull/push blob from/to image registry on external OIDC cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 4.15.z, 4.16, 4.17
    • Image Registry
    • Critical
    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, the internal image registry would not correctly authenticate users on clusters configured with external OIDC users, making it impossible for users to push or pull images to and from the internal image registry. With this fix, the internal image registry starts using the SelfSubjectReview api, dropping use of the openshift specific user api, which is not available on clusters configured with external OIDC users, making it possible to successfully authenticate with the image registry again.
      Show
      Previously, the internal image registry would not correctly authenticate users on clusters configured with external OIDC users, making it impossible for users to push or pull images to and from the internal image registry. With this fix, the internal image registry starts using the SelfSubjectReview api, dropping use of the openshift specific user api, which is not available on clusters configured with external OIDC users, making it possible to successfully authenticate with the image registry again.
    • Bug Fix
    • In Progress

      Description of problem:

      user.openshift.io and oauth.openshift.io APIs are not unavailable in external oidc cluster, that conducts all the common pull/push blob from/to image registry failed.

      Version-Release number of selected component (if applicable):

      4.15.15

      How reproducible:

      always

      Steps to Reproduce:

      1.Create a ROSA HCP cluster which configured external oidc users
      2.Push data to image registry under a project
      oc new-project wxj1
      oc new-build httpd~https://github.com/openshift/httpd-ex.git 
      3.
      

      Actual results:

      $ oc logs -f build/httpd-ex-1
      Cloning "https://github.com/openshift/httpd-ex.git" ...	Commit:	1edee8f58c0889616304cf34659f074fda33678c (Update httpd.json)	Author:	Petr Hracek <phracek@redhat.com>	Date:	Wed Jun 5 13:00:09 2024 +0200time="2024-06-12T09:55:13Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"I0612 09:55:13.306937       1 defaults.go:112] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on].Caching blobs under "/var/cache/blobs".Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/httpd@sha256:765aa645587f34e310e49db7cdc97e82d34122adb0b604eea891e0f98050aa77...Warning: Pull failed, retrying in 5s ...Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/httpd@sha256:765aa645587f34e310e49db7cdc97e82d34122adb0b604eea891e0f98050aa77...Warning: Pull failed, retrying in 5s ...Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/httpd@sha256:765aa645587f34e310e49db7cdc97e82d34122adb0b604eea891e0f98050aa77...Warning: Pull failed, retrying in 5s ...error: build error: After retrying 2 times, Pull image still failed due to error: unauthorized: unable to validate token: NotFound
      
      
      oc logs -f deploy/image-registry -n openshift-image-registry
      
      time="2024-06-12T09:55:13.36003996Z" level=error msg="invalid token: the server could not find the requested resource (get users.user.openshift.io ~)" go.version="go1.20.12 X:strictfipsruntime" http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=0c380b81-99d4-4118-8de3-407706e8767c http.request.method=GET http.request.remoteaddr="10.130.0.35:50550" http.request.uri="/openshift/token?account=serviceaccount&scope=repository%3Aopenshift%2Fhttpd%3Apull" http.request.useragent="containers/5.28.0 (github.com/containers/image)"

      Expected results:

      Should pull/push blob from/to image registry on external oidc cluster

      Additional info:

       

            fmissi Flavian Missi
            rh-ee-xiuwang XiuJuan Wang
            xiujuan wang xiujuan wang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: