-
Bug
-
Resolution: Done-Errata
-
Normal
-
4.10
-
Important
-
No
-
Rejected
-
False
-
This is a clone of issue OCPBUGS-19675. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13044. The following is the description of the original issue:
—
Description of problem:
During cluster installations/upgrades with an imageContentSourcePolicy in place but with access to quay.io, the ICSP is not honored to pull the machine-os-content image from a private registry.
Version-Release number of selected component (if applicable):
$ oc logs -n openshift-machine-config-operator ds/machine-config-daemon -c machine-config-daemon|head -1 Found 6 pods, using pod/machine-config-daemon-znknf I0503 10:53:00.925942 2377 start.go:112] Version: v4.12.0-202304070941.p0.g87fedee.assembly.stream-dirty (87fedee690ae487f8ae044ac416000172c9576a5)
How reproducible:
100% in clusters with ICSP configured BUT with access to quay.io
Steps to Reproduce:
1. Create mirror repo: $ cat <<EOF > /tmp/isc.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 archiveSize: 4 storageConfig: registry: imageURL: quay.example.com/mirror/oc-mirror-metadata skipTLS: true mirror: platform: channels: - name: stable-4.12 type: ocp minVersion: 4.12.13 graph: true EOF $ oc mirror --dest-skip-tls --config=/tmp/isc.yaml docker://quay.example.com/mirror/oc-mirror-metadata <...> info: Mirroring completed in 2m27.91s (138.6MB/s) Writing image mapping to oc-mirror-workspace/results-1683104229/mapping.txt Writing UpdateService manifests to oc-mirror-workspace/results-1683104229 Writing ICSP manifests to oc-mirror-workspace/results-1683104229 2. Confirm machine-os-content digest: $ oc adm release info 4.12.13 -o jsonpath='{.references.spec.tags[?(@.name=="machine-os-content")].from}'|jq { "kind": "DockerImage", "name": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1660c8086ff85e569e10b3bc9db344e1e1f7530581d742ad98b670a81477b1b" } $ oc adm release info 4.12.14 -o jsonpath='{.references.spec.tags[?(@.name=="machine-os-content")].from}'|jq { "kind": "DockerImage", "name": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed68d04d720a83366626a11297a4f3c5761c0b44d02ef66fe4cbcc70a6854563" } 3. Create 4.12.13 cluster with ICSP at install time: $ grep imageContentSources -A6 ./install-config.yaml imageContentSources: - mirrors: - quay.example.com/mirror/oc-mirror-metadata/openshift/release source: quay.io/openshift-release-dev/ocp-v4.0-art-dev - mirrors: - quay.example.com/mirror/oc-mirror-metadata/openshift/release-images source: quay.io/openshift-release-dev/ocp-release
Actual results:
1. After the installation is completed, no pulls for a166 (4.12.13-x86_64-machine-os-content) are logged in the Quay usage logs whereas e.g. digest 22d2 (4.12.13-x86_64-machine-os-images) are reported to be pulled from the mirror. 2. After upgrading to 4.12.14 no pulls for ed68 (4.12.14-x86_64-machine-os-content) are logged in the mirror-registry while the image was pulled as part of `oc image extract` in the machine-config-daemon: [core@master-1 ~]$ sudo less /var/log/pods/openshift-machine-config-operator_machine-config-daemon-7fnjz_e2a3de54-1355-44f9-a516-2f89d6c6ab8f/machine-config-daemon/0.log 2023-05-03T10:51:43.308996195+00:00 stderr F I0503 10:51:43.308932 11290 run.go:19] Running: nice -- ionice -c 3 oc image extract -v 10 --path /:/run/mco-extensions/os-extensions-content-4035545447 --registry- config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad48fe01f3e82584197797ce2151eecdfdcce67ae1096f06412e5ace416f66ce 2023-05-03T10:51:43.418211869+00:00 stderr F I0503 10:51:43.418008 184455 client_mirrored.go:174] Attempting to connect to quay.io/openshift-release-dev/ocp-v4.0-art-dev 2023-05-03T10:51:43.418211869+00:00 stderr F I0503 10:51:43.418174 184455 round_trippers.go:466] curl -v -XGET -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/31aa3e8" 'https://quay.io/v2/' 2023-05-03T10:51:43.419618513+00:00 stderr F I0503 10:51:43.419517 184455 round_trippers.go:495] HTTP Trace: DNS Lookup for quay.io resolved to [{34.206.15.82 } {54.209.210.231 } {52.5.187.29 } {52.3.168.193 } {52.21.36.23 } {50.17.122.58 } {44.194.68.221 } {34.194.241.136 } {2600:1f18:483:cf01:ebba:a861:1150:e245 } {2600:1f18:483:cf02:40f9:477f:ea6b:8a2b } {2600:1f18:483:cf02:8601:2257:9919:cd9e } {2600:1f18:483:cf01 :8212:fcdc:2a2a:50a7 } {2600:1f18:483:cf00:915d:9d2f:fc1f:40a7 } {2600:1f18:483:cf02:7a8b:1901:f1cf:3ab3 } {2600:1f18:483:cf00:27e2:dfeb:a6c7:c4db } {2600:1f18:483:cf01:ca3f:d96e:196c:7867 }] 2023-05-03T10:51:43.429298245+00:00 stderr F I0503 10:51:43.429151 184455 round_trippers.go:510] HTTP Trace: Dial to tcp:34.206.15.82:443 succeed
Expected results:
All images are pulled from the location as configured in the ICSP.
Additional info:
- clones
-
OCPBUGS-19675 machine-config-operator does not honor ICSP when fetching machine-os-content
- Closed
- is blocked by
-
OCPBUGS-19675 machine-config-operator does not honor ICSP when fetching machine-os-content
- Closed
- links to
-
RHBA-2023:5677 OpenShift Container Platform 4.12.z bug fix update