Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-19779

machine-config-operator does not honor ICSP when fetching machine-os-content


    • Important
    • No
    • Rejected
    • False
    • Hide



      This is a clone of issue OCPBUGS-19675. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-13044. The following is the description of the original issue:

      Description of problem:

      During cluster installations/upgrades with an imageContentSourcePolicy in place but with access to quay.io, the ICSP is not honored to pull the machine-os-content image from a private registry.

      Version-Release number of selected component (if applicable):

      $ oc logs -n openshift-machine-config-operator ds/machine-config-daemon -c machine-config-daemon|head -1
      Found 6 pods, using pod/machine-config-daemon-znknf
      I0503 10:53:00.925942    2377 start.go:112] Version: v4.12.0-202304070941.p0.g87fedee.assembly.stream-dirty (87fedee690ae487f8ae044ac416000172c9576a5)

      How reproducible:

      100% in clusters with ICSP configured BUT with access to quay.io

      Steps to Reproduce:

      1. Create mirror repo:
      $ cat <<EOF > /tmp/isc.yaml                                                    
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      archiveSize: 4
          imageURL: quay.example.com/mirror/oc-mirror-metadata
          skipTLS: true
          - name: stable-4.12
            type: ocp
            minVersion: 4.12.13
          graph: true
      $ oc mirror --dest-skip-tls  --config=/tmp/isc.yaml docker://quay.example.com/mirror/oc-mirror-metadata
      info: Mirroring completed in 2m27.91s (138.6MB/s)
      Writing image mapping to oc-mirror-workspace/results-1683104229/mapping.txt
      Writing UpdateService manifests to oc-mirror-workspace/results-1683104229
      Writing ICSP manifests to oc-mirror-workspace/results-1683104229
      2. Confirm machine-os-content digest:
      $ oc adm release info 4.12.13 -o jsonpath='{.references.spec.tags[?(@.name=="machine-os-content")].from}'|jq
        "kind": "DockerImage",
        "name": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1660c8086ff85e569e10b3bc9db344e1e1f7530581d742ad98b670a81477b1b"
      $ oc adm release info 4.12.14 -o jsonpath='{.references.spec.tags[?(@.name=="machine-os-content")].from}'|jq
        "kind": "DockerImage",
        "name": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed68d04d720a83366626a11297a4f3c5761c0b44d02ef66fe4cbcc70a6854563"
      3. Create 4.12.13 cluster with ICSP at install time:
      $ grep imageContentSources -A6 ./install-config.yaml
        - mirrors:
          - quay.example.com/mirror/oc-mirror-metadata/openshift/release
          source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
        - mirrors:
          - quay.example.com/mirror/oc-mirror-metadata/openshift/release-images
          source: quay.io/openshift-release-dev/ocp-release

      Actual results:

      1. After the installation is completed, no pulls for a166 (4.12.13-x86_64-machine-os-content) are logged in the Quay usage logs whereas e.g. digest 22d2 (4.12.13-x86_64-machine-os-images) are reported to be pulled from the mirror. 
      2. After upgrading to 4.12.14 no pulls for ed68 (4.12.14-x86_64-machine-os-content) are logged in the mirror-registry while the image was pulled as part of `oc image extract` in the machine-config-daemon:
      [core@master-1 ~]$ sudo less /var/log/pods/openshift-machine-config-operator_machine-config-daemon-7fnjz_e2a3de54-1355-44f9-a516-2f89d6c6ab8f/machine-config-daemon/0.log                        2023-05-03T10:51:43.308996195+00:00 stderr F I0503 10:51:43.308932   11290 run.go:19] Running: nice -- ionice -c 3 oc image extract -v 10 --path /:/run/mco-extensions/os-extensions-content-4035545447 --registry- config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad48fe01f3e82584197797ce2151eecdfdcce67ae1096f06412e5ace416f66ce 2023-05-03T10:51:43.418211869+00:00 stderr F I0503 10:51:43.418008  184455 client_mirrored.go:174] Attempting to connect to quay.io/openshift-release-dev/ocp-v4.0-art-dev 2023-05-03T10:51:43.418211869+00:00 stderr F I0503 10:51:43.418174  184455 round_trippers.go:466] curl -v -XGET  -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/31aa3e8" 'https://quay.io/v2/' 2023-05-03T10:51:43.419618513+00:00 stderr F I0503 10:51:43.419517  184455 round_trippers.go:495] HTTP Trace: DNS Lookup for quay.io resolved to [{ } { } { } { }  { } { } { } { } {2600:1f18:483:cf01:ebba:a861:1150:e245 } {2600:1f18:483:cf02:40f9:477f:ea6b:8a2b } {2600:1f18:483:cf02:8601:2257:9919:cd9e } {2600:1f18:483:cf01 :8212:fcdc:2a2a:50a7 } {2600:1f18:483:cf00:915d:9d2f:fc1f:40a7 } {2600:1f18:483:cf02:7a8b:1901:f1cf:3ab3 } {2600:1f18:483:cf00:27e2:dfeb:a6c7:c4db } {2600:1f18:483:cf01:ca3f:d96e:196c:7867 }] 2023-05-03T10:51:43.429298245+00:00 stderr F I0503 10:51:43.429151  184455 round_trippers.go:510] HTTP Trace: Dial to tcp: succeed 

      Expected results:

      All images are pulled from the location as configured in the ICSP.

      Additional info:


            team-mco Team MCO
            openshift-crt-jira-prow OpenShift Prow Bot
            Sergio Regidor de la Rosa Sergio Regidor de la Rosa
            0 Vote for this issue
            6 Start watching this issue