Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-22392

OSUS does not parse namespaced registry authentication entries

XMLWordPrintable

    • Moderate
    • No
    • 5
    • OTA 249, OTA 250, OTA 255
    • 3
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      When authenticating to a registry containing release images, OSUS uses the <host>:<port> tuple extracted from a dockerconfigjson but does not consider any specific URI (namespace) in the entry.

      Version-Release number of selected component (if applicable):

      $ oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.13.18   True        False         3h39m   Cluster version is 4.13.18
      

      How reproducible:

      100%

      Steps to Reproduce:

      1. mirror 2 separate imagesets, each with distinct top-level namespace (/release & /operators):
      
      $ cat is-4.13-release.yaml 
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      storageConfig:
        local:
          path: ./
      mirror:
        platform:
          channels:
          - name: stable-4.13
            type: ocp
          graph: true
      
         
      $ oc mirror --config=./is-4.13-release.yaml docker://mirror.example.com/release
      <..>
      info: Mirroring completed in 2m9.27s (142MB/s)
      Writing image mapping to oc-mirror-workspace/results-1698235682/mapping.txt
      Writing ICSP manifests to oc-mirror-workspace/results-1698235682   
      
      $ cat is-4.13-operators.yaml 
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      storageConfig:
        local:
          path: ./
      mirror:
        operators:
        - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13
          packages:
          - name: serverless-operator
            channels:
            - name: stable
      
      
      $ oc mirror --config=./is-4.13-operators.yaml docker://mirror.example.com/operators
      <..>
      info: Mirroring completed in 2m33.98s (93.32MB/s)
      Rendering catalog image "mirror.example.com/operators/redhat/redhat-operator-index:v4.13" with file-based catalog 
      Writing image mapping to oc-mirror-workspace/results-1698236710/mapping.txt
      Writing CatalogSource manifests to oc-mirror-workspace/results-1698236710
      Writing ICSP manifests to oc-mirror-workspace/results-16982367103. 
      
      2. create 2 distinct robot accounts, one with access to mirror.example.com/operators, the other to mirror.example.com/release
      
      3. ensure namespaced pullsecrets are present:
      $ oc extract secret/pull-secret -n openshift-config --to=-|jq '.auths|keys'
      # .dockerconfigjson
      [
        "cloud.openshift.com",
        "docker.io",
        "mirror.example.com/operators",   <--- account 1
        "mirror.example.com/release",  
        "mirror.example.com/release/operators",   <--- account 2
        "quay.io",
        "registry.connect.redhat.com",
        "registry.redhat.io"
      ]
      
      mirror.example.com/release/operators/redhat/redhat-operator-index:v4.13
      
      
      4. install osus through cli
      
      5. create osus instance
      $ cat <<EOF |oc apply -f -  
      apiVersion: updateservice.operator.openshift.io/v1
      kind: UpdateService
      metadata:
        name: sample
        namespace: openshift-update-service
      spec:
        graphDataImage: mirror.example.com/release/openshift/graph-image:latest
        releases: mirror.example.com/release/openshift/release-images
        replicas: 1
      EOF
      updateservice.updateservice.operator.openshift.io/sample created
      
      6. verify the update service cannot scrape the tags from the release repository:
      
      $ oc logs -n openshift-update-service deployment/sample
      2023-10-25T12:48:53Z WARN  cincinnati::plugins::internal::graph_builder::release_scrape_dockerv2::plugin] Error reading registry credentials from "/var/lib/cincinnati/registry-credentials/.dockerconfigjson". Access to "mirror.example.com" will be unauthenticated: authentication information missing for index mirror.example.com 
      [2023-10-25T12:48:53Z DEBUG graph_builder::graph] graph update triggered
      [2023-10-25T12:48:53Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2'
      [2023-10-25T12:48:53Z DEBUG cincinnati::plugins::internal::graph_builder::release_scrape_dockerv2::registry] registry not authorized, attempting anonymous authorization
      [2023-10-25T12:48:53Z ERROR graph_builder::graph] failed to fetch all release metadata from mirror.example.com/release/openshift/release-images
      [2023-10-25T12:48:53Z ERROR graph_builder::graph] http transport error: HTTP status client error (401 Unauthorized) for url (https://mirror.example.com/v2/release/openshift/release-images/tags/list?n=20)
      
      8. confirm skopeo is able to list tags from the same registry using the dockerconfigjson in the pod:  
      $ oc debug -n openshift-update-service deployment/sample --image=registry.redhat.io/rhel8/skopeo -- skopeo list-tags --authfile /var/lib/cincinnati/registry-credentials/.dockerconfigjson docker://mirror.example.com:8443/release/openshift/release-images
      Defaulting container name to graph-builder.
      {
          "Repository": "mirror.example.com/release/openshift/release-images",
          "Tags": [
              "4.13.17-x86_64"
          ]
      }
      

      Actual results:

      OSUS unable to differentiate between namespaced entries in a dockerconfigjson file.

      Expected results:

      OSUS should be able to authenticate based on the most specific authentication entry (similar to container-tools)

      Additional info:

      When parsing the credentials [1] the application extracts the host:port part of the requested registry [2] and passes this host:port tuple to the library reading the credentials [3] But since the tuple does not contain a uri (namespace), it is not considered when matching the registry.
      
      [1] https://github.com/openshift/cincinnati/blob/dc2abd15d0336318829e6788000ac7418c29c2fc/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/plugin.rs#L115-L126
      [2] https://github.com/openshift/cincinnati/blob/dc2abd15d0336318829e6788000ac7418c29c2fc/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/registry/mod.rs#L169-L184
      [3] https://github.com/openshift/cincinnati/blob/dc2abd15d0336318829e6788000ac7418c29c2fc/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/registry/mod.rs#L195-L204

              pratikam Pratik Mahajan
              rhn-support-bverschu Bram Verschueren
              Jian Li Jian Li
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: