Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-43540

dataImportCron import is failing when template points to an internal registry that requires authentication

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • None
    • Storage Platform
    • None
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • ---
    • ---
    • Storage Core Sprint 256, Storage Core Sprint 259
    • No

      Description of problem:

      Context:
      Customer is in a half disconnected environment, Openshift doesn't have
      direct access to internet but access to an Harbor sitting in a DMZ that
      can "cache" images.
      They want to use Openshift Virtualization (Kubevirt) over Openshift Data Foundation(Rook) as a replacement for vsphere
      
      Use case:
      Create  a VM over a full baremetal Openshift cluster. 
      A Harbor registry is currently the ONLY source for pulling the entire images of Openshift, for ocp4 basic container mirroring, olm (Operator Lifecycle Management) images
      for every operator... All is replicated using oc-mirror using a Runner in
      DMZ that "sync" between Internet and Harbor.
      On this harbor, customer has also synchronized for a previous Kubevirt over RKE2 poc the following images :
      quay.io/containerdisks/centos-stream,centos, ubuntu, fedora
      and also
      quay.io/fedora/fedora-coreos-kubevirt
      
      Problem:
      The imageDigestMirrorSet already "rewrite" everything to pull from harbor, and so customer expects that dataImportCron in Kubevirt to sync these images from the local harbor cache by changing the source in the Kubevirt Operator (hyperconverged).
      also, customer expected that when we create a VM on Openshift console and use as source the "Registry (Create PVC) and put "harbor.domain/quay.io/fedora/fedora-coreos-kubevirt:stable" for example, it works. 
      Except there is an issue with the credentials and customer gets : unauthorized to pull repository.
      It's unclear if this happens when we create a VM from the previously mentioned image or when we try to create a BootableVolume Cron to synchronize this source automatically from Harbor as a DataSource available for creating VM.
      
      
      
      Further investigations:
       podman pull on a different machine using the same creds works.
      since the global pull secrets works for the same harbor, doing a 'oc run harbor.domain/quay.io/fedora/fedora-coreos-kubevirt:stable"  it successfully pulls the image 
      but when it's CDI that tries it it fails everytime.
      
      
      
      Yaml output the job to inspect it :
      
      containers:
        - command:
          - /usr/bin/cdi-source-update-poller -ns default -cron fedora-coreos-kubevirt-import-cron...-url docker://harbor.domain/quay.io/fedora/fedora-coreos-kubevirt:stable
          env:
          - name: INSECURE_TLS
            value: "true"
          image: registry.redhat.io/container-native-virtualization/virt-cdi-importer-rhel9
      
      [...]
       imagePullSecrets
       - name: cdi-cronjob-dockercfg
      
      
      Customer tried to check the cronjob secrets and even overwrite it with a new one container the exact same credentials, still doesn't work.
      
      
      
      The Cluster Configurations for Image operator is also referencing Harbor as allowedRegistries (I also tried to reference it as InsecureRegistries, but doesn't change anything),
      I also followed the docs to add our CA so it trust our self signed CA but
      nothing, I also modified the Kubevirt operator to add under spec.storageImport.insecureRegistries
      our harbor url with 3 entries, one on :5000, one on :443 (we usually use
      this one) and one without any port. Still not working.

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Always

      Steps to Reproduce:

      See above
      

      Actual results:

      unauthorized to pull repository

      Expected results:

      Successful startup of vm

      Additional info:

       

              agilboa@redhat.com Arnon Gilboa
              skhoury@redhat.com Sherine Khoury
              Kevin Alon Goldblatt Kevin Alon Goldblatt
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: