Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-36408

Unable to pull from private registry due to missing certs

XMLWordPrintable

    • Critical
    • No
    • 0
    • WINC - Sprint 256
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      *Cause*: An image pull from a registry with a self-signed CA is required
      *Consequence*: The image pull fails due to the CA not being recognized
      *Fix*: Registry certificates provided to the cluster by the user are loaded into the Windows trust store on each Node
      *Result*: Images can be pulled from registries with self-signed CAs
      Show
      *Cause*: An image pull from a registry with a self-signed CA is required *Consequence*: The image pull fails due to the CA not being recognized *Fix*: Registry certificates provided to the cluster by the user are loaded into the Windows trust store on each Node *Result*: Images can be pulled from registries with self-signed CAs
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-35936. The following is the description of the original issue:

      Description of problem:

      Mirrored disconnected image cannot be pulled from a disconnected registry, image has been successfully mirrored to disconnected registry, yet workloads cannot be created 

      Version-Release number of selected component (if applicable):

      10.16.0-60c5aec

      How reproducible:

      100%

      Steps to Reproduce:

      1. create a disconnected Windows containers cluster
      2. mirror Windows container image mcr.microsoft.com/powershell:lts-nanoserver-ltsc2022, oc image mirror <image> --keep-manifest-list=true --insecure=true --skip-verification
      3. create image tag and image digest
      MIRRORED_IMAGE="${MIRROR_REGISTRY}/powershell"SOURCE_IMAGE="mcr.microsoft.com/powershell"
      SOURCE_IMAGE="mcr.microsoft.com/powershell"
      cat <<EOF | oc apply -f -                                 
      apiVersion: config.openshift.io/v1
      kind: ImageTagMirrorSet
      metadata:
        name: powershell-tag-mirror
      spec:
        imageTagMirrors:
        - source: ${SOURCE_IMAGE}
          mirrors:
          - ${MIRRORED_IMAGE}
      EOF
      
      cat <<EOF | oc apply -f -                                  INT | 17:51:57
      apiVersion: config.openshift.io/v1
      kind: ImageDigestMirrorSet
      metadata:
        name: powershell-digest-mirror
      spec:
        imageDigestMirrors:
        - source: ${SOURCE_IMAGE}
          mirrors:
          - ${MIRRORED_IMAGE}
      EOF
      
      Name:         powershell-mirror
      Namespace:
      Labels:       <none>
      Annotations:  <none>
      API Version:  config.openshift.io/v1
      Kind:         ImageTagMirrorSet
      Metadata:
        Creation Timestamp:  2024-06-23T11:24:50Z
        Generation:          1
        Resource Version:    54488
        UID:                 17c49c1e-04e3-42ca-afec-833b63598c29
      Spec:
        Image Tag Mirrors:
          Mirrors:
            rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell
          Source:  mcr.microsoft.com/powershell
      
      Name:         powershell-digest-mirror
      Namespace:
      Labels:       <none>
      Annotations:  <none>
      API Version:  config.openshift.io/v1
      Kind:         ImageDigestMirrorSet
      Metadata:
        Creation Timestamp:  2024-06-23T12:16:16Z
        Generation:          1
        Resource Version:    79545
        UID:                 b7458a93-1c85-44bb-b8ed-f60db24f7601
      Spec:
        Image Digest Mirrors:
          Mirrors:
            rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell
          Source:  mcr.microsoft.com/powershell
      Events:      <none>
      cat C:\k\containerd\registries\mcr.microsoft.com\hosts.toml
      server = "https://mcr.microsoft.com"[host."https://rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000"]
        capabilities = ["pull", "resolve"]
        ca = "C:\\k\\ca-bundle.crt"
        [host."https://rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000".header]
          authorization = "Basic ZHVtbXk6ZHVtbXk=" 4. Append CA to CM registry-config -n openshift-config 
      # rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000 PS C:\Users\Administrator> cat C:\\k\\ca-bundle.crt
      -----BEGIN CERTIFICATE----- 
      ...
      -----END CERTIFICATE-----
      5. create Windows workload using the following image:
      rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell:lts-nanoserver-ltsc2022  

      Actual results:

      Image cannot be created
      Events:
        Type     Reason     Age                  From               Message
        ----     ------     ----                 ----               -------
        Normal   Scheduled  140m                 default-scheduler  Successfully assigned winc-test/win-webserver-7f6f4f586c-llxt5 to winworker-5vdfm
        Normal   Pulling    138m (x4 over 140m)  kubelet            Pulling image "rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell:lts-nanoserver-ltsc2022"
        Warning  Failed     138m (x4 over 140m)  kubelet            Error: ErrImagePull
        Normal   BackOff    7s (x613 over 140m)  kubelet            Back-off pulling image "rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell:lts-nanoserver-ltsc2022"
      

      Expected results:

      Pulling succesfully from a disconnected registry fails
      
      

      Additional info:

      06-23 13:45:43.915  Running command: oc image mirror "mcr.microsoft.com/powershell:lts-nanoserver-ltsc2022=rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell:lts-nanoserver-ltsc2022" --keep-manifest-list=true --insecure=true --skip-verification
      06-23 13:45:43.916  rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/
      06-23 13:45:43.916    powershell
      06-23 13:45:43.916      blobs:
      06-23 13:45:43.916        mcr.microsoft.com/powershell sha256:29be79fac85e0752ff343bab27b66192d2049cb2da5bcbc72dfab8bbdffcc1f2 1.014KiB
      06-23 13:45:43.916        mcr.microsoft.com/powershell sha256:61ba9509da787a35028ad3606c87a5aa68e6cef9291ceb767595db47873611be 1.014KiB
      06-23 13:45:43.916        mcr.microsoft.com/powershell sha256:947916e0db43525f0b3a35d8c1cda6969bba3fe7b836eca48f658af5272715ef 1.015KiB
      06-23 13:45:43.916        mcr.microsoft.com/powershell sha256:4b5442942dafc1a14f727b63c3eb6f05a300b07db1487d8cc24b27c6d7bfdbd7 1.018KiB
      06-23 13:45:43.917        mcr.microsoft.com/powershell sha256:5e8a0e572a283c8a65a09a5cd07ddf79a1042c0e596cb6cdf77b19970696546e 1.018KiB
      06-23 13:45:43.917        mcr.microsoft.com/powershell sha256:bb88967ce4ee0d476e43d7f7067c4043310821427eefca257e777bb086ccb267 3.984KiB
      06-23 13:45:43.917        mcr.microsoft.com/powershell sha256:b379a8d125a9a775afc9889ba18f0262d08f596e4621e0b2ade67ed2d40ead39 83.86KiB
      06-23 13:45:43.917        mcr.microsoft.com/powershell sha256:e645b2dc4e5a32639efac79779d7b0cb9e558bc69cb0874ea1a25ea14e5aa3b5 112.7KiB
      06-23 13:45:43.917        mcr.microsoft.com/powershell sha256:b3f15c3bed556c018f671b2ceb36a92001ac64b4aefa4002401830b9b3ab718d 103.5MiB
      06-23 13:45:43.917        mcr.microsoft.com/powershell sha256:755fc767289b8847bd0d0d8d75efc308c040140acf2a3426973ba9fbf022c4c0 115.4MiB
      06-23 13:45:43.918      manifests:
      06-23 13:45:43.918        sha256:816b28df3ce39a36d6c6f696a4fd6f7823e09defb2af865f31501e868cb0e082 -> lts-nanoserver-ltsc2022
      06-23 13:45:43.918    stats: shared=0 unique=10 size=219.1MiB ratio=1.00
      06-23 13:45:43.918  
      06-23 13:45:43.918  phase 0:
      06-23 13:45:43.918    rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000 powershell blobs=10 mounts=0 manifests=1 shared=0
      06-23 13:45:43.918  
      06-23 13:45:43.918  info: Planning completed in 440ms
      06-23 13:45:43.918  uploading: rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell sha256:755fc767289b8847bd0d0d8d75efc308c040140acf2a3426973ba9fbf022c4c0 115.4MiB
      06-23 13:45:43.918  uploading: rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell sha256:b379a8d125a9a775afc9889ba18f0262d08f596e4621e0b2ade67ed2d40ead39 83.86KiB
      06-23 13:45:43.919  uploading: rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell sha256:e645b2dc4e5a32639efac79779d7b0cb9e558bc69cb0874ea1a25ea14e5aa3b5 112.7KiB
      06-23 13:45:43.919  uploading: rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell sha256:b3f15c3bed556c018f671b2ceb36a92001ac64b4aefa4002401830b9b3ab718d 103.5MiB
      06-23 13:46:22.573  sha256:816b28df3ce39a36d6c6f696a4fd6f7823e09defb2af865f31501e868cb0e082 rrasouli-2004-bastion.mirror-registry.qe.devcluster.openshift.com:5000/powershell:lts-nanoserver-ltsc2022
      06-23 13:46:22.574  info: Mirroring completed in 44.75s (5.133MB/s)
      

            rh-ee-mankulka Mansi Kulkarni
            openshift-crt-jira-prow OpenShift Prow Bot
            Aharon Rasouli Aharon Rasouli
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: