Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4282

[IR-269]Can't pull sub-manifest image using imagestream of manifest list

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • premerge
    • ImageStreams
    • Critical
    • None
    • Sprint 230, Sprint 231
    • 2
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      When pull image in a container using imagestream of manifest list via pullthrough , the sub-manifest image can't be pulled with error "Unknown desc = copying system image from manifest list: reading signatures: downloading signatures"

      Version-Release number of selected component (if applicable):

      oc version
      Client Version: 4.12.0-0.nightly-2022-11-22-215931
      Kustomize Version: v4.5.7
      Server Version: 4.12.0-0.ci.test-2022-11-30-035930-ci-ln-3mxz61t-latest
      Kubernetes Version: v1.25.0-2585+553373323f102c-dirty

      How reproducible:

      always

      Steps to Reproduce:

      1.Create ImageStreamImport with oci images(Which have include multi arch images)cat <<EOF | oc create -f -
      apiVersion: image.openshift.io/v1
      kind: ImageStreamImport
      metadata:
        name: ociapp
      spec:
        import: true
        images:
        - from:
            kind: DockerImage
            name: quay.io/openshifttest/ociimage:multicarch
          to:
            name: latest
          referencePolicy:
            type: Local
          importPolicy:
            importMode: "PreserveOriginal"
      EOF
      
      
      2. Check the imagestream
      $oc describe is ociapp
      Name:            ociapp
      Namespace:        wxj
      Created:        20 minutes ago
      Labels:            <none>
      Annotations:        openshift.io/image.dockerRepositoryCheck=2022-11-30T06:03:04Z
      Image Repository:    image-registry.openshift-image-registry.svc:5000/wxj/ociapp
      Image Lookup:        local=false
      Unique Images:        1
      Tags:            1latest
        tagged from quay.io/openshifttest/ociimage:multiarch
          prefer registry pullthrough when referencing this tag  * quay.io/openshifttest/ociimage@sha256:d58e3e003ddec723dd14f72164beaa609d24c5e5e366579e23bc8b34b9a58324
            18 minutes ago
      3.Create pod using manifest list imagestream oc run test --image=image-registry.openshift-image-registry.svc:5000/wxj/ociapp:latest --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"seccompProfile":{"type":"RuntimeDefault"}}}}' -- sleep 300
      
      4.Check the pod if is running
      oc get pods
      NAME   READY   STATUS         RESTARTS   AGE
      test   0/1     ErrImagePull   0          33s

      Actual results:

      4.
        Normal   Scheduled       9s    default-scheduler  Successfully assigned wxj/test to ip-10-0-152-87.us-east-2.compute.internal
        Normal   AddedInterface  10s   multus             Add eth0 [10.131.0.27/23] from ovn-kubernetes
        Normal   Pulling         9s    kubelet            Pulling image "image-registry.openshift-image-registry.svc:5000/wxj/ociapp:latest"
        Warning  Failed          8s    kubelet            Failed to pull image "image-registry.openshift-image-registry.svc:5000/wxj/ociapp:latest": rpc error: code = Unknown desc = copying system image from manifest list: reading signatures: downloading signatures for sha256:97923994fdc1c968eed6bdcb64be8e70d5356b88cfab0481cb6b73a4849361b7 in image-registry.openshift-image-registry.svc:5000/wxj/ociapp: error parsing HTTP 404 response body: unexpected end of JSON input: ""
        Warning  Failed          8s    kubelet            Error: ErrImagePull
        Normal   BackOff         7s    kubelet            Back-off pulling image "image-registry.openshift-image-registry.svc:5000/wxj/ociapp:latest"
        Warning  Failed          7s    kubelet            Error: ImagePullBackOff
      
      
      registry log 
      time="2022-11-30T06:06:46.138179499Z" level=info msg="authorized request" go.version=go1.19.2 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=ea4e5812-55bf-4c79-8ac8-8c28db3c5f35 http.request.method=GET http.request.remoteaddr="100.64.0.5:53358" http.request.uri="/extensions/v2/wxj/ociapp/signatures/sha256:97923994fdc1c968eed6bdcb64be8e70d5356b88cfab0481cb6b73a4849361b7" http.request.useragent="cri-o/1.25.1-5.rhaos4.12.git6005903.el8 go/go1.19.2 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wxj:default" vars.digest="sha256:97923994fdc1c968eed6bdcb64be8e70d5356b88cfab0481cb6b73a4849361b7" vars.name=wxj/ociapp
      time="2022-11-30T06:06:46.148685708Z" level=info msg=response go.version=go1.19.2 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=0e78c21a-0ebb-4a2b-936a-784d704ce47b http.request.method=GET http.request.remoteaddr="100.64.0.5:53358" http.request.uri="/extensions/v2/wxj/ociapp/signatures/sha256:97923994fdc1c968eed6bdcb64be8e70d5356b88cfab0481cb6b73a4849361b7" http.request.useragent="cri-o/1.25.1-5.rhaos4.12.git6005903.el8 go/go1.19.2 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=17.362215ms http.response.status=404 http.response.written=0

      Expected results:

      Could pull the sub-manifest image via pullthrough

      Additional info:

      Image registry log
      https://drive.google.com/file/d/1whDdHB8S50MsuM59WLpAnDmnAvOeUubl/view?usp=share_link

            obulatov@redhat.com Oleg Bulatov
            rh-ee-xiuwang XiuJuan Wang
            XiuJuan Wang XiuJuan Wang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: