Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-43231

Cannot pull VM template from private registry with certificate signed by an internal CA

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • CNV v4.14.6
    • Storage Platform
    • None
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • ---
    • ---
    • Important
    • No

      Description of problem:

      In a disconnected environment, we want to import a custom template from our private registry that uses a certificate signed by an internal CA.

      Version-Release number of selected component (if applicable):

      OpenShift Virtualization 4.14.6
      registry.redhat.io/container-native-virtualization/virt-cdi-importer-rhel9@sha256:2554e92cccd77ccd348c0150612847f61b98c5474c58b6b640ca013703f81506

      How reproducible:

      Always

      Steps to Reproduce:

      1. Add a custom dataImportCronTemplates to the HyperConverged:
      
        dataImportCronTemplates:
        - metadata:
            annotations:
              cdi.kubevirt.io/storage.bind.immediate.requested: "true"
            name: my-fedora
          spec:
            managedDataSource: my-fedora
            retentionPolicy: None
            schedule: '* * * * *'
            template:
              spec:
                source:
                  registry:
                    certConfigMap: registry-ca-cert
                    pullMethod: node
                    url: docker://jorti-registry.sbr-virt.gsslab.brq2.redhat.com:5000/fedora:latest
                storage:
                  resources:
                    requests:
                      storage: 30Gi 
      
      2. Create the registry-ca-cert configmap in the openshift-cnv and openshift-virtualization-os-images with the CA certificate:
      
      apiVersion: v1
      data:
        ca.pem: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
      kind: ConfigMap
      metadata:
        name: registry-ca-cert
        namespace: openshift-cnv
      
      
      

      Actual results:

      In the openshift-cnv namespace, the pods initial-job-my-fedora-XXXXX-XXX fail to validate the registry cert:
      
      $ oc logs -f my-fedora-1443f369-28649216-jzgpl
      I0621 06:56:01.045821       1 registry-datasource.go:176] Copying proxy certs
      2024/06/21 06:56:01 Ignore common certificate dir: open /proxycerts/: no such file or directory
      I0621 06:56:01.046148       1 transport.go:228] Inspecting image from 'docker://jorti-registry.sbr-virt.gsslab.brq2.redhat.com:5000/fedora:latest'
      E0621 06:56:01.087402       1 transport.go:78] Could not create image reference: pinging container registry jorti-registry.sbr-virt.gsslab.brq2.redhat.com:5000: Get "https://jorti-registry.sbr-virt.gsslab.brq2.redhat.com:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
      2024/06/21 06:56:01 Failed to get image digest: Could not create image reference: pinging container registry jorti-registry.sbr-virt.gsslab.brq2.redhat.com:5000: Get "https://jorti-registry.sbr-virt.gsslab.brq2.redhat.com:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
      
      
      
      $ oc describe pod my-fedora-1443f369-28649216-jzgpl
      Name:             my-fedora-1443f369-28649216-jzgpl
      Namespace:        openshift-cnv
      Priority:         0
      Service Account:  cdi-cronjob
      Node:             jorti-cluster03-worker-2/10.37.201.145
      Start Time:       Fri, 21 Jun 2024 08:56:00 +0200
      Labels:           batch.kubernetes.io/controller-uid=33f8fcca-29b2-48fb-8e63-0695259d6aa2
                        batch.kubernetes.io/job-name=my-fedora-1443f369-28649216
                        controller-uid=33f8fcca-29b2-48fb-8e63-0695259d6aa2
                        job-name=my-fedora-1443f369-28649216
      Annotations:      k8s.ovn.org/pod-networks:
                          {"default":{"ip_addresses":["10.128.2.35/23"],"mac_address":"0a:58:0a:80:02:23","gateway_ips":["10.128.2.1"],"routes":[{"dest":"10.128.0.0...
                        k8s.v1.cni.cncf.io/network-status:
                          [{
                              "name": "ovn-kubernetes",
                              "interface": "eth0",
                              "ips": [
                                  "10.128.2.35"
                              ],
                              "mac": "0a:58:0a:80:02:23",
                              "default": true,
                              "dns": {}
                          }]
                        openshift.io/scc: containerized-data-importer
                        seccomp.security.alpha.kubernetes.io/pod: runtime/default
      Status:           Failed
      SeccompProfile:   RuntimeDefault
      IP:               10.128.2.35
      IPs:
        IP:           10.128.2.35
      Controlled By:  Job/my-fedora-1443f369-28649216
      Containers:
        cdi-source-update-poller:
          Container ID:    cri-o://48f991c7d778f8295a3d923b0def088b07067d55368da52c30e98d713c5e3c2a
          Image:           registry.redhat.io/container-native-virtualization/virt-cdi-importer-rhel9@sha256:2554e92cccd77ccd348c0150612847f61b98c5474c58b6b640ca013703f81506
          Image ID:        registry.redhat.io/container-native-virtualization/virt-cdi-importer-rhel9@sha256:2554e92cccd77ccd348c0150612847f61b98c5474c58b6b640ca013703f81506
          Port:            <none>
          Host Port:       <none>
          SeccompProfile:  RuntimeDefault
          Command:
            /usr/bin/cdi-source-update-poller
            -ns
            openshift-virtualization-os-images
            -cron
            my-fedora
            -url
            docker://jorti-registry.sbr-virt.gsslab.brq2.redhat.com:5000/fedora:latest
            -certdir
            /certs
          State:          Terminated
            Reason:       Error
            Exit Code:    1
            Started:      Fri, 21 Jun 2024 08:56:00 +0200
            Finished:     Fri, 21 Jun 2024 08:56:01 +0200
          Ready:          False
          Restart Count:  0
          Environment:
            http_proxy:   
            https_proxy:  
            no_proxy:     
          Mounts:
            /certs from cdi-cert-vol (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-77zqr (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        cdi-cert-vol:
          Type:      ConfigMap (a volume populated by a ConfigMap)
          Name:      registry-ca-cert
          Optional:  false
        kube-api-access-77zqr:
          Type:                    Projected (a volume that contains injected data from multiple sources)
          TokenExpirationSeconds:  3607
          ConfigMapName:           kube-root-ca.crt
          ConfigMapOptional:       <nil>
          DownwardAPI:             true
          ConfigMapName:           openshift-service-ca.crt
          ConfigMapOptional:       <nil>
      QoS Class:                   BestEffort
      Node-Selectors:              <none>
      Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
      Events:
        Type    Reason          Age   From               Message
        ----    ------          ----  ----               -------
        Normal  Scheduled       10s   default-scheduler  Successfully assigned openshift-cnv/my-fedora-1443f369-28649216-jzgpl to jorti-cluster03-worker-2
        Normal  AddedInterface  11s   multus             Add eth0 [10.128.2.35/23] from ovn-kubernetes
        Normal  Pulled          11s   kubelet            Container image "registry.redhat.io/container-native-virtualization/virt-cdi-importer-rhel9@sha256:2554e92cccd77ccd348c0150612847f61b98c5474c58b6b640ca013703f81506" already present on machine
        Normal  Created         11s   kubelet            Created container cdi-source-update-poller
        Normal  Started         10s   kubelet            Started container cdi-source-update-poller

      Expected results:

      A successful pull

      Additional info:

       

              rhn-support-awels Alexander Wels
              rhn-support-jortialc Juan Orti
              Kevin Alon Goldblatt Kevin Alon Goldblatt
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: