Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-10273

[BUG] ABI image creation would not point to mirror registry.

XMLWordPrintable

    • Moderate
    • No
    • Agent Sprint 233, Sprint 235
    • 2
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      The ABI install step to create the ISO wouldn't point to the local registry to grab the rhcos live ISO.
      
      --------
      [root@bastion ~]# openshift-install version
      openshift-install 4.12.6
      built from commit fd9e75e61946e79de4e5d9959c7d681cc0271043
      release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
      release architecture amd64
      [root@bastion ~]# oc version
      Client Version: 4.12.6
      Kustomize Version: v4.5.7
      [root@bastion ~]#
      --------
      
      The results to create ISO would be as,
      
      [root@bastion ~]# openshift-install --dir socp4 agent create image --log-level=debug
      DEBUG OpenShift Installer 4.12.6
      DEBUG Built from commit fd9e75e61946e79de4e5d9959c7d681cc0271043
      DEBUG Fetching Agent Installer ISO...
      DEBUG Loading Agent Installer ISO...
      DEBUG   Loading Agent Installer Ignition...
      DEBUG     Loading Agent Manifests...
      DEBUG       Loading Agent PullSecret...
      DEBUG         Loading Install Config...
      DEBUG         Using Install Config loaded from target directory
      DEBUG       Loading InfraEnv Config...
      DEBUG         Loading Install Config...
      DEBUG         Loading Agent Config...
      DEBUG         Using Agent Config loaded from target directory
      DEBUG       Loading NMState Config...
      DEBUG         Loading Agent Config...
      DEBUG         Loading Install Config...
      DEBUG       Loading AgentClusterInstall Config...
      DEBUG         Loading Install Config...
      DEBUG       Loading ClusterDeployment Config...
      DEBUG         Loading Install Config...
      DEBUG       Loading ClusterImageSet Config...
      DEBUG         Loading Release Image Pull Spec...
      DEBUG         Loading Install Config...
      DEBUG     Loading Extra Manifests...
      DEBUG     Loading Certificate (kube-apiserver-lb-signer)...
      DEBUG     Loading Certificate (kube-apiserver-localhost-signer)...
      DEBUG     Loading Certificate (kube-apiserver-service-network-signer)...
      DEBUG     Loading Certificate (admin-kubeconfig-signer)...
      DEBUG     Loading Kubeadmin Password...
      DEBUG     Loading Agent Config...
      DEBUG     Loading Mirror Registries Config...
      DEBUG       Loading Install Config...
      DEBUG       Loading Release Image Pull Spec...
      DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
      DEBUG     Loading Mirror Registries Certificate File...
      DEBUG       Loading Install Config...
      DEBUG   Loading BaseIso Image...
      DEBUG     Loading Agent Manifests...
      DEBUG     Loading Install Config...
      DEBUG     Loading Mirror Registries Config...
      DEBUG   Fetching Agent Installer Ignition...
      DEBUG     Fetching Agent Manifests...
      DEBUG       Fetching Agent PullSecret...
      DEBUG         Fetching Install Config...
      DEBUG         Reusing previously-fetched Install Config
      DEBUG       Generating Agent PullSecret...
      DEBUG       Fetching InfraEnv Config...
      DEBUG         Fetching Install Config...
      DEBUG         Reusing previously-fetched Install Config
      DEBUG         Fetching Agent Config...
      DEBUG         Reusing previously-fetched Agent Config
      DEBUG       Generating InfraEnv Config...
      DEBUG       Fetching NMState Config...
      DEBUG         Fetching Agent Config...
      DEBUG         Reusing previously-fetched Agent Config
      DEBUG         Fetching Install Config...
      DEBUG         Reusing previously-fetched Install Config
      DEBUG       Generating NMState Config...
      DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d6:c7:08
      DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d6:c7:09
      DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:b8:80
      DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:b8:81
      DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:ba:7c
      DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:ba:7d
      DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d6:c7:10
      DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d6:c7:11
      DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:bc:1c
      DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:bc:1d
      DEBUG       Fetching AgentClusterInstall Config...
      DEBUG         Fetching Install Config...
      DEBUG         Reusing previously-fetched Install Config
      DEBUG       Generating AgentClusterInstall Config...
      DEBUG       Fetching ClusterDeployment Config...
      DEBUG         Fetching Install Config...
      DEBUG         Reusing previously-fetched Install Config
      DEBUG       Generating ClusterDeployment Config...
      DEBUG       Fetching ClusterImageSet Config...
      DEBUG         Fetching Release Image Pull Spec...
      DEBUG         Generating Release Image Pull Spec...
      DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
      DEBUG         Fetching Install Config...
      DEBUG         Reusing previously-fetched Install Config
      DEBUG       Generating ClusterImageSet Config...
      DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
      DEBUG     Generating Agent Manifests...
      DEBUG     Fetching Extra Manifests...
      DEBUG     Generating Extra Manifests...
      DEBUG     Fetching Certificate (kube-apiserver-lb-signer)...
      DEBUG     Generating Certificate (kube-apiserver-lb-signer)...
      DEBUG     Fetching Certificate (kube-apiserver-localhost-signer)...
      DEBUG     Generating Certificate (kube-apiserver-localhost-signer)...
      DEBUG     Fetching Certificate (kube-apiserver-service-network-signer)...
      DEBUG     Generating Certificate (kube-apiserver-service-network-signer)...
      DEBUG     Fetching Certificate (admin-kubeconfig-signer)...
      DEBUG     Generating Certificate (admin-kubeconfig-signer)...
      DEBUG     Fetching Kubeadmin Password...
      DEBUG     Generating Kubeadmin Password...
      DEBUG     Fetching Agent Config...
      DEBUG     Reusing previously-fetched Agent Config
      DEBUG     Fetching Mirror Registries Config...
      DEBUG       Fetching Install Config...
      DEBUG       Reusing previously-fetched Install Config
      DEBUG       Fetching Release Image Pull Spec...
      DEBUG       Reusing previously-fetched Release Image Pull Spec
      DEBUG     Generating Mirror Registries Config...
      WARNING The ImageContentSources configuration in install-config.yaml should have at-least one source field matching the releaseImage value quay.io/openshift-release-dev/ocp-release@sha256
      DEBUG     Fetching Mirror Registries Certificate File...
      DEBUG       Fetching Install Config...
      DEBUG       Reusing previously-fetched Install Config
      DEBUG     Generating Mirror Registries Certificate File...
      DEBUG   Generating Agent Installer Ignition...
      DEBUG RendezvousIP from the AgentConfig 10.40.1.55
      INFO The rendezvous host IP (node0 IP) is 10.40.1.55
      DEBUG Generated random infra-env id 1912d5c5-feb8-4035-8dd5-0a74e2f64c6b
      DEBUG   Fetching BaseIso Image...
      DEBUG     Fetching Agent Manifests...
      DEBUG     Reusing previously-fetched Agent Manifests
      DEBUG     Fetching Install Config...
      DEBUG     Reusing previously-fetched Install Config
      DEBUG     Fetching Mirror Registries Config...
      DEBUG     Reusing previously-fetched Mirror Registries Config
      DEBUG   Generating BaseIso Image...
      INFO Extracting base ISO from release payload
      DEBUG Using mirror configuration
      DEBUG Fetching image from OCP release (oc adm release info --image-for=machine-os-images --insecure=true --icsp-file=/tmp/icsp-file2798914892 quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b)
      DEBUG extracting /coreos/coreos-x86_64.iso to /root/.cache/agent/image_cache, oc image extract --path /coreos/coreos-x86_64.iso:/root/.cache/agent/image_cache --confirm --icsp-file=/tmp/icsp-file3074929307 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9d36b729016727cb4acf85f858ed437671747b3d3df0afca02e52d2f68ce19
      WARNING Failed to extract base ISO from release payload - check registry configuration
      INFO Downloading base ISO
      DEBUG Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.12/builds/412.86.202301311551-0/x86_64/rhcos-412.86.202301311551-0-live.x86_64.iso'------------------------------------------------------------------------------->download iso from the internet.
      DEBUG The file was found in cache: /root/.cache/agent/image_cache/rhcos-412.86.202301311551-0-live.x86_64.iso
      DEBUG Using base ISO image /root/.cache/agent/image_cache/rhcos-412.86.202301311551-0-live.x86_64.iso
      DEBUG Generating Agent Installer ISO...
      DEBUG initDisk(): start
      DEBUG initDisk(): regular file
      DEBUG trying fat32
      DEBUG fat32 failed: Error reading MS-DOS Boot Sector: Could not read FAT32 BIOS Parameter Block from boot sector: Could not read embedded DOS 3.31 BPB: Error reading embedded DOS 2.0 BPB: Invalid sector size 37008 provided in DOS 2.0 BPB. Must be 512
      DEBUG trying iso9660 with physical block size 0
      INFO Consuming Agent Config from target directory
      DEBUG Purging asset "Agent Config" from disk
      INFO Consuming Install Config from target directory
      DEBUG Purging asset "Install Config" from disk
      DEBUG Fetching Kubeconfig Admin Client...
      DEBUG Loading Kubeconfig Admin Client...
      DEBUG   Loading Certificate (admin-kubeconfig-client)...
      DEBUG     Loading Certificate (admin-kubeconfig-signer)...
      DEBUG   Loading Certificate (kube-apiserver-complete-server-ca-bundle)...
      DEBUG     Loading Certificate (kube-apiserver-localhost-ca-bundle)...
      DEBUG       Loading Certificate (kube-apiserver-localhost-signer)...
      DEBUG     Loading Certificate (kube-apiserver-service-network-ca-bundle)...
      DEBUG       Loading Certificate (kube-apiserver-service-network-signer)...
      DEBUG     Loading Certificate (kube-apiserver-lb-ca-bundle)...
      DEBUG       Loading Certificate (kube-apiserver-lb-signer)...
      DEBUG   Loading ClusterDeployment Config...
      DEBUG   Fetching Certificate (admin-kubeconfig-client)...
      DEBUG     Fetching Certificate (admin-kubeconfig-signer)...
      DEBUG     Reusing previously-fetched Certificate (admin-kubeconfig-signer)
      DEBUG   Generating Certificate (admin-kubeconfig-client)...
      DEBUG   Fetching Certificate (kube-apiserver-complete-server-ca-bundle)...
      DEBUG     Fetching Certificate (kube-apiserver-localhost-ca-bundle)...
      DEBUG       Fetching Certificate (kube-apiserver-localhost-signer)...
      DEBUG       Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)
      DEBUG     Generating Certificate (kube-apiserver-localhost-ca-bundle)...
      DEBUG     Fetching Certificate (kube-apiserver-service-network-ca-bundle)...
      DEBUG       Fetching Certificate (kube-apiserver-service-network-signer)...
      DEBUG       Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)
      DEBUG     Generating Certificate (kube-apiserver-service-network-ca-bundle)...
      DEBUG     Fetching Certificate (kube-apiserver-lb-ca-bundle)...
      DEBUG       Fetching Certificate (kube-apiserver-lb-signer)...
      DEBUG       Reusing previously-fetched Certificate (kube-apiserver-lb-signer)
      DEBUG     Generating Certificate (kube-apiserver-lb-ca-bundle)...
      DEBUG   Generating Certificate (kube-apiserver-complete-server-ca-bundle)...
      DEBUG   Fetching ClusterDeployment Config...
      DEBUG   Reusing previously-fetched ClusterDeployment Config
      DEBUG Generating Kubeconfig Admin Client...
      DEBUG Fetching Kubeadmin Password...
      DEBUG Reusing previously-fetched Kubeadmin Password
      [root@bastion ~]#
      -------------------------------
      
      
      Here we can see the https://rhcos.mirror.openshift.com is used to grab the ISO. Whereas the logs shows failure with the payload as,
      
      ---------------
      DEBUG       Reusing previously-fetched Release Image Pull Spec
      DEBUG     Generating Mirror Registries Config...
      WARNING The ImageContentSources configuration in install-config.yaml should have at-least one source field matching the releaseImage value quay.io/openshift-release-dev/ocp-release@sha256
      DEBUG     Fetching Mirror Registries Certificate File...
      DEBUG       Fetching Install Config...
      ---------------
      
      
      
      At this stage in install-config.yaml we have,
      
      ---------------
      imageContentSources:
        - mirrors:
          - ocpregistry.xyz.xz.com:5000/ocp412/openshift/release
          source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
        - mirrors:
          - ocpregistry.xyz.xz.com:5000/ocp412/openshift/release-images
          source: quay.io/openshift-release-dev/ocp-release
      ---------------
      
      So likely the "openshift-install" isn't pointing to the correct payload as it seems and then gets directed to the mirrored site rather registry.
      
      The registry used is created with oc-mirror plugin from documetation,
      
      https://docs.openshift.com/container-platform/4.12/installing/disconnected_install/installing-mirroring-disconnected.html#installing-mirroring-disconnected
      
      which do not specify something like "oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" which I believe marks the correct payload to the openshift-install tool as per the other way to create registry ( https://docs.openshift.com/container-platform/4.12/installing/disconnected_install/installing-mirroring-installation-images.html#installing-mirroring-installation-images )
      
      
      Note: oc-mirror is used create the registry as starting with version 4.11, only the file-based catalog is supported and OLM is need to customer.

       

      Version-Release number of selected component (if applicable):

      4.12

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create a local mirror registry with oc-mirror
      2. Add the "imageContentSources" to install-config.yaml
      3. run the command to generate the ISO with debug logs are enabled and it can be seen the ISO is fetched from mirrored site.
      
      Note: If the system is not having internet access then the command will hung for ever.
      

      Actual results:

      The RHCOS is fetched from mirror site and not from payload.

      Expected results:

      It shall fetch the ISO from local registry / payload shall match to local registry.

      Additional info:

      We tried to use "oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" but it failed with an error as unknown manifests so we believe either we are missing on the configuration to export wrt the registry or as oc-mirror documentation doesn't have it mentioned it could be not supported or not work. Though ready to test any alternative suggested.

              bfournie@redhat.com Robert Fournier
              rhn-support-pkhedeka Parikshit Khedekar
              Manoj Hans Manoj Hans
              Anil Dhingra
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: