Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-13044

machine-config-operator does not honor ICSP when fetching machine-os-content

    XMLWordPrintable

Details

    • Important
    • MCO Sprint 242
    • 1
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Machine Config Operator (MCO) leveraged the `oc image extract` command to pull images during updates but the `ImageContentSourcePolicy` (ICSP) object was not respected when pulling those images. With this update, the MCO now uses the `podman pull` command internally and images are pulled from the location as configured in the ICSP. (https://issues.redhat.com/browse/OCPBUGS-13044[*OCPBUGS-13044*])
      Show
      * Previously, the Machine Config Operator (MCO) leveraged the `oc image extract` command to pull images during updates but the `ImageContentSourcePolicy` (ICSP) object was not respected when pulling those images. With this update, the MCO now uses the `podman pull` command internally and images are pulled from the location as configured in the ICSP. ( https://issues.redhat.com/browse/OCPBUGS-13044 [* OCPBUGS-13044 *])
    • Bug Fix
    • Done

    Description

      Description of problem:

      During cluster installations/upgrades with an imageContentSourcePolicy in place but with access to quay.io, the ICSP is not honored to pull the machine-os-content image from a private registry.

      Version-Release number of selected component (if applicable):

      $ oc logs -n openshift-machine-config-operator ds/machine-config-daemon -c machine-config-daemon|head -1
      Found 6 pods, using pod/machine-config-daemon-znknf
      I0503 10:53:00.925942    2377 start.go:112] Version: v4.12.0-202304070941.p0.g87fedee.assembly.stream-dirty (87fedee690ae487f8ae044ac416000172c9576a5)
      

      How reproducible:

      100% in clusters with ICSP configured BUT with access to quay.io

      Steps to Reproduce:

      1. Create mirror repo:
      $ cat <<EOF > /tmp/isc.yaml                                                    
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      archiveSize: 4
      storageConfig:
        registry:
          imageURL: quay.example.com/mirror/oc-mirror-metadata
          skipTLS: true
      mirror:
        platform:
          channels:
          - name: stable-4.12
            type: ocp
            minVersion: 4.12.13
          graph: true
      EOF
      $ oc mirror --dest-skip-tls  --config=/tmp/isc.yaml docker://quay.example.com/mirror/oc-mirror-metadata
      <...>
      info: Mirroring completed in 2m27.91s (138.6MB/s)
      Writing image mapping to oc-mirror-workspace/results-1683104229/mapping.txt
      Writing UpdateService manifests to oc-mirror-workspace/results-1683104229
      Writing ICSP manifests to oc-mirror-workspace/results-1683104229
      
      2. Confirm machine-os-content digest:
      $ oc adm release info 4.12.13 -o jsonpath='{.references.spec.tags[?(@.name=="machine-os-content")].from}'|jq
      {
        "kind": "DockerImage",
        "name": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1660c8086ff85e569e10b3bc9db344e1e1f7530581d742ad98b670a81477b1b"
      }
      $ oc adm release info 4.12.14 -o jsonpath='{.references.spec.tags[?(@.name=="machine-os-content")].from}'|jq
      {
        "kind": "DockerImage",
        "name": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed68d04d720a83366626a11297a4f3c5761c0b44d02ef66fe4cbcc70a6854563"
      }
      
      3. Create 4.12.13 cluster with ICSP at install time:
      $ grep imageContentSources -A6 ./install-config.yaml
      imageContentSources:
        - mirrors:
          - quay.example.com/mirror/oc-mirror-metadata/openshift/release
          source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
        - mirrors:
          - quay.example.com/mirror/oc-mirror-metadata/openshift/release-images
          source: quay.io/openshift-release-dev/ocp-release
      
      
      

      Actual results:

      1. After the installation is completed, no pulls for a166 (4.12.13-x86_64-machine-os-content) are logged in the Quay usage logs whereas e.g. digest 22d2 (4.12.13-x86_64-machine-os-images) are reported to be pulled from the mirror. 
      
      2. After upgrading to 4.12.14 no pulls for ed68 (4.12.14-x86_64-machine-os-content) are logged in the mirror-registry while the image was pulled as part of `oc image extract` in the machine-config-daemon:
      
      [core@master-1 ~]$ sudo less /var/log/pods/openshift-machine-config-operator_machine-config-daemon-7fnjz_e2a3de54-1355-44f9-a516-2f89d6c6ab8f/machine-config-daemon/0.log                        2023-05-03T10:51:43.308996195+00:00 stderr F I0503 10:51:43.308932   11290 run.go:19] Running: nice -- ionice -c 3 oc image extract -v 10 --path /:/run/mco-extensions/os-extensions-content-4035545447 --registry- config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad48fe01f3e82584197797ce2151eecdfdcce67ae1096f06412e5ace416f66ce 2023-05-03T10:51:43.418211869+00:00 stderr F I0503 10:51:43.418008  184455 client_mirrored.go:174] Attempting to connect to quay.io/openshift-release-dev/ocp-v4.0-art-dev 2023-05-03T10:51:43.418211869+00:00 stderr F I0503 10:51:43.418174  184455 round_trippers.go:466] curl -v -XGET  -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/31aa3e8" 'https://quay.io/v2/' 2023-05-03T10:51:43.419618513+00:00 stderr F I0503 10:51:43.419517  184455 round_trippers.go:495] HTTP Trace: DNS Lookup for quay.io resolved to [{34.206.15.82 } {54.209.210.231 } {52.5.187.29 } {52.3.168.193 }  {52.21.36.23 } {50.17.122.58 } {44.194.68.221 } {34.194.241.136 } {2600:1f18:483:cf01:ebba:a861:1150:e245 } {2600:1f18:483:cf02:40f9:477f:ea6b:8a2b } {2600:1f18:483:cf02:8601:2257:9919:cd9e } {2600:1f18:483:cf01 :8212:fcdc:2a2a:50a7 } {2600:1f18:483:cf00:915d:9d2f:fc1f:40a7 } {2600:1f18:483:cf02:7a8b:1901:f1cf:3ab3 } {2600:1f18:483:cf00:27e2:dfeb:a6c7:c4db } {2600:1f18:483:cf01:ca3f:d96e:196c:7867 }] 2023-05-03T10:51:43.429298245+00:00 stderr F I0503 10:51:43.429151  184455 round_trippers.go:510] HTTP Trace: Dial to tcp:34.206.15.82:443 succeed 

      Expected results:

      All images are pulled from the location as configured in the ICSP.

      Additional info:

       

      Attachments

        Issue Links

          Activity

            People

              rhn-engineering-skumari Sinny Kumari
              rhn-support-bverschu Bram Verschueren
              Sergio Regidor de la Rosa Sergio Regidor de la Rosa
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: