Major Description of problem:
When I execute the following two tag commands in a row on OCP 4.14.0-ec.3, Multi-Arch: oc tag $IMAGE@$DIGEST_MANIFEST test-1:tag-manifest sleep 0 oc tag $IMAGE@$DIGEST_MANIFEST test-1:tag-manifest-preserve-original --import-mode=PreserveOriginal Then wrong data is written to the .image.dockerImageMetadata record. If there is a delay between these two commands, e.g. sleep 5, then the image.dockerImageMetadata contains correct data.
Version-Release number of selected component (if applicable):
How reproducible:
Run the below script and you see the error. If you change the SLEEP_TIME=5, then the script passes. No problem.
Steps to Reproduce:
#!/usr/bin/env bash
set -e
SLEEP_TIME=0 # Test will fail, when sleep time is 0, use delay of 3 sec or more to pass this test
IMAGE="quay.io/podman/hello"
podman pull $IMAGE:latest
DIGEST_MANIFEST=$(podman inspect quay.io/podman/hello:latest | jq -r '.[0].Digest')
oc new-project "ir-test-001"
oc create imagestream test-1
oc import-image test-1 --from="${IMAGE}@${DIGEST_MANIFEST}" --import-mode='PreserveOriginal'
oc tag $IMAGE@$DIGEST_MANIFEST test-1:tag-manifest
sleep "${SLEEP_TIME}"
oc tag $IMAGE@$DIGEST_MANIFEST test-1:tag-manifest-preserve-original --import-mode=PreserveOriginal
sleep 5
[[ $(oc get istag test-1:tag-manifest-preserve-original -o json | jq -r '.image.dockerImageMetadata.Architecture') == "null" ]] && echo "pass: tag-manifest-preserve-original has no architecture" || echo "fail: tag-preserve-original has architecture and should not"
Actual results:
fail: tag-preserve-original has architecture and should not oc get istag test-1:tag-manifest-preserve-original -o json | jq -r '.image.dockerImageMetadata.Architecture' amd64
Expected results:
pass: tag-manifest-preserve-original has no architecture oc get istag test-1:tag-manifest-preserve-original -o json | jq -r '.image.dockerImageMetadata.Architecture' null
Additional info:
This was tested with OC command on x86_64
[OCPBUGS-16514] OCP 4.14 | Execution of two oc tag commands in a row, creates wrong .image.dockerImageMetadata
Validate this bug on 4.15.0-0.nightly-2023-09-22-023217 cluster
Import two images with the same digest but different import mode, could get the correct import mode digest $oc get istag NAME IMAGE REFERENCE UPDATED test-1:tag-manifest quay.io/podman/hello@sha256:6a461b416dbe29f7d63153ecf2fa7a5f147de971fc23f988de98e683349071fe 6 minutes ago test-1:tag-manifest-preserve-original quay.io/podman/hello@sha256:075975296016084fc66b59c35c9d4504765d95aadcd5469f28d2b75750348fc5 6 minutes ago $oc describe istag test-1:tag-manifest Image Name: sha256:6a461b416dbe29f7d63153ecf2fa7a5f147de971fc23f988de98e683349071fe Docker Image: quay.io/podman/hello@sha256:6a461b416dbe29f7d63153ecf2fa7a5f147de971fc23f988de98e683349071fe Name: sha256:6a461b416dbe29f7d63153ecf2fa7a5f147de971fc23f988de98e683349071fe Created: 6 minutes ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 28.86kB in 1 layers Layers: 27.53kB sha256:d08b40be68780d583e8c127f10228743e3e1beb520f987c0e32f4ef0c0ce8020 Image Created: 3 weeks ago Author: <none> Arch: amd64 $oc describe istag test-1:tag-manifest-preserve-originalImage Name: sha256:075975296016084fc66b59c35c9d4504765d95aadcd5469f28d2b75750348fc5Docker Image: quay.io/podman/hello@sha256:075975296016084fc66b59c35c9d4504765d95aadcd5469f28d2b75750348fc5Name: sha256:075975296016084fc66b59c35c9d4504765d95aadcd5469f28d2b75750348fc5 Created: 7 minutes ago Image Created: 7 minutes agoAuthor: <none> Arch: <none> Manifests: linux/amd64 sha256:6a461b416dbe29f7d63153ecf2fa7a5f147de971fc23f988de98e683349071fe linux/arm64 sha256:aae49df2db8b242a5a0e80b727b40536b250d8c71417baf9a6cfc68e8850b92b linux/s390x sha256:09dcf2c2ae9e368f21d916998bba0c301ac31314c6a0408e0a39d0514d584d39 linux/ppc64le sha256:d7b8967f4ec0ad9d5a38a9d6175df9f0cd0b36d2656dd7615b726559075f2fbb
Flavian Missi
added a comment - Hi psundara@redhat.com , sort of. I was able to reproduce the problem with the suggested commands, but you can alternatively use the following steps to reproduce it:
1. create the namespace: `oc new project ir-test-001`
2. create a yaml with the following contents, call it isi.yaml:
```
apiVersion: image.openshift.io/v1
kind: ImageStreamImport
metadata:
name: test-1
namespace: ir-test-001
spec:
import: true
images:
- from:
kind: DockerImage
name: quay.io/podman/hello@sha256:afda668e706a451073f23642c3b3cf0a8988284c35121b1ff42288c6235532b4
to:
name: tag-manifest
referencePolicy:
type: Source
- from:
kind: DockerImage
name: quay.io/podman/hello@sha256:afda668e706a451073f23642c3b3cf0a8988284c35121b1ff42288c6235532b4
to:
name: tag-manifest-preserve-original
referencePolicy:
type: Source
importPolicy:
importMode: PreserveOriginal
```
3. create the image stream by importing the images above: `oc create -y isi.yaml`
4. note that the result image stream `.status.` images point to the same digest (`oc get is test-1 -ojson|jq '.status.tags[].items[].image'`), which refers to the amd64 image(`sha256:3381e3a704e8067b5925522d420483e2ce7a966bc1ef7b3008b90539560fc8f4`), even though the second image to import specified the import mode "PreserveOriginal"
Let me know if you have further questions.
Flavian Missi
added a comment - Hi psundara@redhat.com , sort of. I was able to reproduce the problem with the suggested commands, but you can alternatively use the following steps to reproduce it:
1. create the namespace: `oc new project ir-test-001`
2. create a yaml with the following contents, call it isi.yaml:
```
apiVersion: image.openshift.io/v1
kind: ImageStreamImport
metadata:
name: test-1
namespace: ir-test-001
spec:
import: true
images:
- from:
kind: DockerImage
name: quay.io/podman/hello@sha256:afda668e706a451073f23642c3b3cf0a8988284c35121b1ff42288c6235532b4
to:
name: tag-manifest
referencePolicy:
type: Source
- from:
kind: DockerImage
name: quay.io/podman/hello@sha256:afda668e706a451073f23642c3b3cf0a8988284c35121b1ff42288c6235532b4
to:
name: tag-manifest-preserve-original
referencePolicy:
type: Source
importPolicy:
importMode: PreserveOriginal
```
3. create the image stream by importing the images above: `oc create -y isi.yaml`
4. note that the result image stream `.status.` images point to the same digest (`oc get is test-1 -ojson|jq '.status.tags[].items[].image'`), which refers to the amd64 image(`sha256:3381e3a704e8067b5925522d420483e2ce7a966bc1ef7b3008b90539560fc8f4`), even though the second image to import specified the import mode "PreserveOriginal"
Let me know if you have further questions.
Prashanth Sundararaman
added a comment - That's great news! thanks Flavian! i wasn't able tp reproduce this - so is there a specific scenario when the failure happens?
Prashanth Sundararaman
added a comment - That's great news! thanks Flavian! i wasn't able tp reproduce this - so is there a specific scenario when the failure happens? Flavian Missi
added a comment - I've narrowed down the problem to the openshift-apiserver import code. A local cache key is to blame. I'm putting together a fix, should be able to roll it out some time next week. Unless we decide otherwise this will be out on 4.15.
Flavian Missi
added a comment - I've narrowed down the problem to the openshift-apiserver import code. A local cache key is to blame. I'm putting together a fix, should be able to roll it out some time next week. Unless we decide otherwise this will be out on 4.15. Flavian Missi
added a comment - Hi folks,
I was able to reproduce this on an single arch AWS cluster. I think the problem is on the openshift-apiserver tag code, but I haven't been able to identify the exact spot yet. I think it's a combination of import mode and that both tags are using the same digest as source image but as said, I'm not sure where the issue lies yet.
The code is here for those who might be curious: https://github.com/openshift/openshift-apiserver/blob/master/pkg/image/apiserver/registry/imagestreamtag/rest.go
I'll move this to the ImageStreams component as I think oc is not at fault.
Flavian Missi
added a comment - Hi folks,
I was able to reproduce this on an single arch AWS cluster. I think the problem is on the openshift-apiserver tag code, but I haven't been able to identify the exact spot yet. I think it's a combination of import mode and that both tags are using the same digest as source image but as said, I'm not sure where the issue lies yet.
The code is here for those who might be curious: https://github.com/openshift/openshift-apiserver/blob/master/pkg/image/apiserver/registry/imagestreamtag/rest.go
I'll move this to the ImageStreams component as I think oc is not at fault. Hello Prashanth,
Dominik and I can still recreate the described issue on the latest 4.14.0-ec.4 build.
How do you want to continue on this defect ? Should we schedule a meeting to show you the issue ?
Prashanth Sundararaman
added a comment - To be clear - i'm not seeing the original failure that is in the description of this bug where the tag has been created but it doesn't seem to do the right thing when adding preserveOriginal importMode. What I'm seeing is the tag not yet created because it hasn't been processed by the apiserver yet.
Prashanth Sundararaman
added a comment - To be clear - i'm not seeing the original failure that is in the description of this bug where the tag has been created but it doesn't seem to do the right thing when adding preserveOriginal importMode. What I'm seeing is the tag not yet created because it hasn't been processed by the apiserver yet. Hello Prashanth,
I am also using x86 Control nodes, but I do not get any error !
I do not see an error like, "tagging hasn't been completed' on my test system.
Even it would be related to a internal cluster networking issue, I expect to see an error.
If I do not define a delay between both tag commands, I have a "data integrity problem" with the metadata and no error is reported to the user. I consider it as an major problem.
Prashanth Sundararaman
added a comment - I tried this script on my x86 laptop connecting to an aws cluster and it works fine. when I remove the "sleep 5" at the end before the check I get an error because the tagging hasn't been completed yet so that's ok. I would say this is an issue specific to the environment. Given that a sleep is needed, I would think there is some slowness in the internal cluster networking.
Prashanth Sundararaman
added a comment - I tried this script on my x86 laptop connecting to an aws cluster and it works fine. when I remove the "sleep 5" at the end before the check I get an error because the tagging hasn't been completed yet so that's ok. I would say this is an issue specific to the environment. Given that a sleep is needed, I would think there is some slowness in the internal cluster networking. 
Klaus verified the solution -> bug fixed -> close