-
Bug
-
Resolution: Cannot Reproduce
-
Normal
-
None
-
4.12.0
Description of problem:
The ABI install step to create the ISO wouldn't point to the local registry to grab the rhcos live ISO. -------- [root@bastion ~]# openshift-install version openshift-install 4.12.6 built from commit fd9e75e61946e79de4e5d9959c7d681cc0271043 release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b release architecture amd64 [root@bastion ~]# oc version Client Version: 4.12.6 Kustomize Version: v4.5.7 [root@bastion ~]# -------- The results to create ISO would be as, [root@bastion ~]# openshift-install --dir socp4 agent create image --log-level=debug DEBUG OpenShift Installer 4.12.6 DEBUG Built from commit fd9e75e61946e79de4e5d9959c7d681cc0271043 DEBUG Fetching Agent Installer ISO... DEBUG Loading Agent Installer ISO... DEBUG Loading Agent Installer Ignition... DEBUG Loading Agent Manifests... DEBUG Loading Agent PullSecret... DEBUG Loading Install Config... DEBUG Using Install Config loaded from target directory DEBUG Loading InfraEnv Config... DEBUG Loading Install Config... DEBUG Loading Agent Config... DEBUG Using Agent Config loaded from target directory DEBUG Loading NMState Config... DEBUG Loading Agent Config... DEBUG Loading Install Config... DEBUG Loading AgentClusterInstall Config... DEBUG Loading Install Config... DEBUG Loading ClusterDeployment Config... DEBUG Loading Install Config... DEBUG Loading ClusterImageSet Config... DEBUG Loading Release Image Pull Spec... DEBUG Loading Install Config... DEBUG Loading Extra Manifests... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Kubeadmin Password... DEBUG Loading Agent Config... DEBUG Loading Mirror Registries Config... DEBUG Loading Install Config... DEBUG Loading Release Image Pull Spec... DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b DEBUG Loading Mirror Registries Certificate File... DEBUG Loading Install Config... DEBUG Loading BaseIso Image... DEBUG Loading Agent Manifests... DEBUG Loading Install Config... DEBUG Loading Mirror Registries Config... DEBUG Fetching Agent Installer Ignition... DEBUG Fetching Agent Manifests... DEBUG Fetching Agent PullSecret... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Agent PullSecret... DEBUG Fetching InfraEnv Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Agent Config... DEBUG Reusing previously-fetched Agent Config DEBUG Generating InfraEnv Config... DEBUG Fetching NMState Config... DEBUG Fetching Agent Config... DEBUG Reusing previously-fetched Agent Config DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating NMState Config... DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d6:c7:08 DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d6:c7:09 DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:b8:80 DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:b8:81 DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:ba:7c DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:ba:7d DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d6:c7:10 DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d6:c7:11 DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:bc:1c DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:bc:1d DEBUG Fetching AgentClusterInstall Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating AgentClusterInstall Config... DEBUG Fetching ClusterDeployment Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating ClusterDeployment Config... DEBUG Fetching ClusterImageSet Config... DEBUG Fetching Release Image Pull Spec... DEBUG Generating Release Image Pull Spec... DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating ClusterImageSet Config... DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b DEBUG Generating Agent Manifests... DEBUG Fetching Extra Manifests... DEBUG Generating Extra Manifests... DEBUG Fetching Certificate (kube-apiserver-lb-signer)... DEBUG Generating Certificate (kube-apiserver-lb-signer)... DEBUG Fetching Certificate (kube-apiserver-localhost-signer)... DEBUG Generating Certificate (kube-apiserver-localhost-signer)... DEBUG Fetching Certificate (kube-apiserver-service-network-signer)... DEBUG Generating Certificate (kube-apiserver-service-network-signer)... DEBUG Fetching Certificate (admin-kubeconfig-signer)... DEBUG Generating Certificate (admin-kubeconfig-signer)... DEBUG Fetching Kubeadmin Password... DEBUG Generating Kubeadmin Password... DEBUG Fetching Agent Config... DEBUG Reusing previously-fetched Agent Config DEBUG Fetching Mirror Registries Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Release Image Pull Spec... DEBUG Reusing previously-fetched Release Image Pull Spec DEBUG Generating Mirror Registries Config... WARNING The ImageContentSources configuration in install-config.yaml should have at-least one source field matching the releaseImage value quay.io/openshift-release-dev/ocp-release@sha256 DEBUG Fetching Mirror Registries Certificate File... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Mirror Registries Certificate File... DEBUG Generating Agent Installer Ignition... DEBUG RendezvousIP from the AgentConfig 10.40.1.55 INFO The rendezvous host IP (node0 IP) is 10.40.1.55 DEBUG Generated random infra-env id 1912d5c5-feb8-4035-8dd5-0a74e2f64c6b DEBUG Fetching BaseIso Image... DEBUG Fetching Agent Manifests... DEBUG Reusing previously-fetched Agent Manifests DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Mirror Registries Config... DEBUG Reusing previously-fetched Mirror Registries Config DEBUG Generating BaseIso Image... INFO Extracting base ISO from release payload DEBUG Using mirror configuration DEBUG Fetching image from OCP release (oc adm release info --image-for=machine-os-images --insecure=true --icsp-file=/tmp/icsp-file2798914892 quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b) DEBUG extracting /coreos/coreos-x86_64.iso to /root/.cache/agent/image_cache, oc image extract --path /coreos/coreos-x86_64.iso:/root/.cache/agent/image_cache --confirm --icsp-file=/tmp/icsp-file3074929307 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9d36b729016727cb4acf85f858ed437671747b3d3df0afca02e52d2f68ce19 WARNING Failed to extract base ISO from release payload - check registry configuration INFO Downloading base ISO DEBUG Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.12/builds/412.86.202301311551-0/x86_64/rhcos-412.86.202301311551-0-live.x86_64.iso'------------------------------------------------------------------------------->download iso from the internet. DEBUG The file was found in cache: /root/.cache/agent/image_cache/rhcos-412.86.202301311551-0-live.x86_64.iso DEBUG Using base ISO image /root/.cache/agent/image_cache/rhcos-412.86.202301311551-0-live.x86_64.iso DEBUG Generating Agent Installer ISO... DEBUG initDisk(): start DEBUG initDisk(): regular file DEBUG trying fat32 DEBUG fat32 failed: Error reading MS-DOS Boot Sector: Could not read FAT32 BIOS Parameter Block from boot sector: Could not read embedded DOS 3.31 BPB: Error reading embedded DOS 2.0 BPB: Invalid sector size 37008 provided in DOS 2.0 BPB. Must be 512 DEBUG trying iso9660 with physical block size 0 INFO Consuming Agent Config from target directory DEBUG Purging asset "Agent Config" from disk INFO Consuming Install Config from target directory DEBUG Purging asset "Install Config" from disk DEBUG Fetching Kubeconfig Admin Client... DEBUG Loading Kubeconfig Admin Client... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading ClusterDeployment Config... DEBUG Fetching Certificate (admin-kubeconfig-client)... DEBUG Fetching Certificate (admin-kubeconfig-signer)... DEBUG Reusing previously-fetched Certificate (admin-kubeconfig-signer) DEBUG Generating Certificate (admin-kubeconfig-client)... DEBUG Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-localhost-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) DEBUG Generating Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-service-network-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) DEBUG Generating Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-lb-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-lb-signer) DEBUG Generating Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Generating Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Fetching ClusterDeployment Config... DEBUG Reusing previously-fetched ClusterDeployment Config DEBUG Generating Kubeconfig Admin Client... DEBUG Fetching Kubeadmin Password... DEBUG Reusing previously-fetched Kubeadmin Password [root@bastion ~]# ------------------------------- Here we can see the https://rhcos.mirror.openshift.com is used to grab the ISO. Whereas the logs shows failure with the payload as, --------------- DEBUG Reusing previously-fetched Release Image Pull Spec DEBUG Generating Mirror Registries Config... WARNING The ImageContentSources configuration in install-config.yaml should have at-least one source field matching the releaseImage value quay.io/openshift-release-dev/ocp-release@sha256 DEBUG Fetching Mirror Registries Certificate File... DEBUG Fetching Install Config... --------------- At this stage in install-config.yaml we have, --------------- imageContentSources: - mirrors: - ocpregistry.xyz.xz.com:5000/ocp412/openshift/release source: quay.io/openshift-release-dev/ocp-v4.0-art-dev - mirrors: - ocpregistry.xyz.xz.com:5000/ocp412/openshift/release-images source: quay.io/openshift-release-dev/ocp-release --------------- So likely the "openshift-install" isn't pointing to the correct payload as it seems and then gets directed to the mirrored site rather registry. The registry used is created with oc-mirror plugin from documetation, https://docs.openshift.com/container-platform/4.12/installing/disconnected_install/installing-mirroring-disconnected.html#installing-mirroring-disconnected which do not specify something like "oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" which I believe marks the correct payload to the openshift-install tool as per the other way to create registry ( https://docs.openshift.com/container-platform/4.12/installing/disconnected_install/installing-mirroring-installation-images.html#installing-mirroring-installation-images ) Note: oc-mirror is used create the registry as starting with version 4.11, only the file-based catalog is supported and OLM is need to customer.
Version-Release number of selected component (if applicable):
4.12
How reproducible:
Always
Steps to Reproduce:
1. Create a local mirror registry with oc-mirror 2. Add the "imageContentSources" to install-config.yaml 3. run the command to generate the ISO with debug logs are enabled and it can be seen the ISO is fetched from mirrored site. Note: If the system is not having internet access then the command will hung for ever.
Actual results:
The RHCOS is fetched from mirror site and not from payload.
Expected results:
It shall fetch the ISO from local registry / payload shall match to local registry.
Additional info:
We tried to use "oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" but it failed with an error as unknown manifests so we believe either we are missing on the configuration to export wrt the registry or as oc-mirror documentation doesn't have it mentioned it could be not supported or not work. Though ready to test any alternative suggested.
- relates to
-
OCPBUGS-10810 Add example for oc-mirror to set up ImageContentSources
- Closed