Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-50660

oc adm release new layers vnd.docker on vnd.oci base image

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.19.0
    • oc
    • None
    • None
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Given an image composed of OCI layers / mediaType "application/vnd.oci.image.manifest.v1+json", "oc" will layer on a docker layer to the image during "oc adm release new ... --keep-manifest-list". This creates and invalid image.

      Version-Release number of selected component (if applicable):

          4.19

      How reproducible:

          100%

      Steps to Reproduce:

          1. Create a new release image using a CVO base image of media type "application/vnd.oci.image.manifest.v1+json" like  quay.io/openshift-release-dev/ocp-v4.0-art-dev:sha256-e37ecf2773a70ac05687df126fd4b1eaef4f6fd0702d45a734897fcd3a091702 
      
      
      For example: 
      $ oc adm release new --from-image-stream 4.19-konflux-art-latest --to-image-base=quay.io/openshift-release-dev/ocp-v4.0-art-dev:sha256-e37ecf2773a70ac05687df126fd4b1eaef4f6fd0702d45a734897fcd3a091702 --to-image quay.io/jupierce/test:oc-bug --keep-manifest-list --reference-mode=source
      ...
      
      ...
      error: failed to push image quay.io/jupierce/test:oc-bug: error appending image sha256:e37ecf2773a70ac05687df126fd4b1eaef4f6fd0702d45a734897fcd3a091702: unable to convert the image to a compatible schema version: manifest invalid: manifest invalid
      
      
      
        2. You will not be able to push the image to quay.io because it is an invalid combination of layers and mediaType. If you push the image to a OpenShift cluster image registry, the push will work, but tools like skopeo will error out because of the invalid combination of layers. "oc image info" can read the image metadata and will look like (notice the last layer is not like the others):
      
      $ oc image info registry.ci.openshift.org/ocp/konflux-release:4.19.0-0.konflux-nightly-2025-02-12-191052 -o=json | jq .layers
      [
        {
          "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
          "digest": "sha256:25c75c34b2e2b68ba9245d9cddeb6b8a0887371ed30744064f85241a75704d87",
          "size": 79262296
        },
        {
          "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
          "digest": "sha256:25b65d192ceaca221582b3e7ae607535cafaf1fff7d38baac83b90ddef1b5c89",
          "size": 44423936
        },
        {
          "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
          "digest": "sha256:e4ac9b66d625701eb4e2e1684737c564331edd920fc28966716892936754d406",
          "size": 9042306
        },
        {
          "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
          "digest": "sha256:cf00f715c3c951101c2925c292655ca18ecb47611708261b7204709989bbe6d6",
          "size": 43954381
        },
        {
          "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
          "digest": "sha256:412bcab2afa4c8262ad89ccb795af4e48fcdc168ff15324674bab453a8a44c91",
          "size": 1720789
        }
      ]
      
      

      Actual results:

      "oc" creates an invalid container image unusable by any standard container runtime or tool (skopeo, podman, ..).    

      Expected results:

      "oc" will preserve the oci layers and add a compatible layer.

      Additional info:

      If keep-manifest-list is *not* specified, oc produces a different image format which can be pushed to quay.io successfully:
      
        "mediaType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
        "layers": [
          {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:25c75c34b2e2b68ba9245d9cddeb6b8a0887371ed30744064f85241a75704d87"
          },
          {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:25b65d192ceaca221582b3e7ae607535cafaf1fff7d38baac83b90ddef1b5c89"
          },
          {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:e4ac9b66d625701eb4e2e1684737c564331edd920fc28966716892936754d406"
          },
          {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:cf00f715c3c951101c2925c292655ca18ecb47611708261b7204709989bbe6d6"
          },
          {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:4e2fcbdaf422a95d26c7c641d6e863196d500b6384643e26cabfcb2546938f7e"
          }
        ],
      
      HOWEVER, this is because of a conversion behind the scenes when pushing to quay.io. That conversion does not happen when pushing to a docker distribution registry. 
      
      1. oc adm release new ... --manifest-list=false --to-image=registry.ci.openshift.org/jupierce:test results in an invalid image according to skopeo/podman.
      
      2. oc adm release new ... --manifest-list=false --to-image=quay.io/jupierce/test:tests results in a valid image.
      
      3. mirroring from registry.ci.openshift.org/jupierce:test to quay.io/jupierce/test:tests  results in a valid quay.io image WITH A CHANGED manifest/digest, because quay is doing a conversion.

       

              aos-workloads-staff Workloads Team Bot Account
              jupierce Justin Pierce
              Ying Zhou Ying Zhou
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: