-
Bug
-
Resolution: Cannot Reproduce
-
Normal
-
None
-
4.12.0
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
No
-
None
-
Rejected
-
Agent Sprint 233, Sprint 235
-
2
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
The ABI install step to create the ISO wouldn't point to the local registry to grab the rhcos live ISO.
--------
[root@bastion ~]# openshift-install version
openshift-install 4.12.6
built from commit fd9e75e61946e79de4e5d9959c7d681cc0271043
release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
release architecture amd64
[root@bastion ~]# oc version
Client Version: 4.12.6
Kustomize Version: v4.5.7
[root@bastion ~]#
--------
The results to create ISO would be as,
[root@bastion ~]# openshift-install --dir socp4 agent create image --log-level=debug
DEBUG OpenShift Installer 4.12.6
DEBUG Built from commit fd9e75e61946e79de4e5d9959c7d681cc0271043
DEBUG Fetching Agent Installer ISO...
DEBUG Loading Agent Installer ISO...
DEBUG Loading Agent Installer Ignition...
DEBUG Loading Agent Manifests...
DEBUG Loading Agent PullSecret...
DEBUG Loading Install Config...
DEBUG Using Install Config loaded from target directory
DEBUG Loading InfraEnv Config...
DEBUG Loading Install Config...
DEBUG Loading Agent Config...
DEBUG Using Agent Config loaded from target directory
DEBUG Loading NMState Config...
DEBUG Loading Agent Config...
DEBUG Loading Install Config...
DEBUG Loading AgentClusterInstall Config...
DEBUG Loading Install Config...
DEBUG Loading ClusterDeployment Config...
DEBUG Loading Install Config...
DEBUG Loading ClusterImageSet Config...
DEBUG Loading Release Image Pull Spec...
DEBUG Loading Install Config...
DEBUG Loading Extra Manifests...
DEBUG Loading Certificate (kube-apiserver-lb-signer)...
DEBUG Loading Certificate (kube-apiserver-localhost-signer)...
DEBUG Loading Certificate (kube-apiserver-service-network-signer)...
DEBUG Loading Certificate (admin-kubeconfig-signer)...
DEBUG Loading Kubeadmin Password...
DEBUG Loading Agent Config...
DEBUG Loading Mirror Registries Config...
DEBUG Loading Install Config...
DEBUG Loading Release Image Pull Spec...
DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
DEBUG Loading Mirror Registries Certificate File...
DEBUG Loading Install Config...
DEBUG Loading BaseIso Image...
DEBUG Loading Agent Manifests...
DEBUG Loading Install Config...
DEBUG Loading Mirror Registries Config...
DEBUG Fetching Agent Installer Ignition...
DEBUG Fetching Agent Manifests...
DEBUG Fetching Agent PullSecret...
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Generating Agent PullSecret...
DEBUG Fetching InfraEnv Config...
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Fetching Agent Config...
DEBUG Reusing previously-fetched Agent Config
DEBUG Generating InfraEnv Config...
DEBUG Fetching NMState Config...
DEBUG Fetching Agent Config...
DEBUG Reusing previously-fetched Agent Config
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Generating NMState Config...
DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d6:c7:08
DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d6:c7:09
DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:b8:80
DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:b8:81
DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:ba:7c
DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:ba:7d
DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d6:c7:10
DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d6:c7:11
DEBUG adding MAC interface map to host static network config - Name: ens14f0 MacAddress:b4:96:91:d8:bc:1c
DEBUG adding MAC interface map to host static network config - Name: ens14f1 MacAddress:b4:96:91:d8:bc:1d
DEBUG Fetching AgentClusterInstall Config...
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Generating AgentClusterInstall Config...
DEBUG Fetching ClusterDeployment Config...
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Generating ClusterDeployment Config...
DEBUG Fetching ClusterImageSet Config...
DEBUG Fetching Release Image Pull Spec...
DEBUG Generating Release Image Pull Spec...
DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Generating ClusterImageSet Config...
DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b
DEBUG Generating Agent Manifests...
DEBUG Fetching Extra Manifests...
DEBUG Generating Extra Manifests...
DEBUG Fetching Certificate (kube-apiserver-lb-signer)...
DEBUG Generating Certificate (kube-apiserver-lb-signer)...
DEBUG Fetching Certificate (kube-apiserver-localhost-signer)...
DEBUG Generating Certificate (kube-apiserver-localhost-signer)...
DEBUG Fetching Certificate (kube-apiserver-service-network-signer)...
DEBUG Generating Certificate (kube-apiserver-service-network-signer)...
DEBUG Fetching Certificate (admin-kubeconfig-signer)...
DEBUG Generating Certificate (admin-kubeconfig-signer)...
DEBUG Fetching Kubeadmin Password...
DEBUG Generating Kubeadmin Password...
DEBUG Fetching Agent Config...
DEBUG Reusing previously-fetched Agent Config
DEBUG Fetching Mirror Registries Config...
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Fetching Release Image Pull Spec...
DEBUG Reusing previously-fetched Release Image Pull Spec
DEBUG Generating Mirror Registries Config...
WARNING The ImageContentSources configuration in install-config.yaml should have at-least one source field matching the releaseImage value quay.io/openshift-release-dev/ocp-release@sha256
DEBUG Fetching Mirror Registries Certificate File...
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Generating Mirror Registries Certificate File...
DEBUG Generating Agent Installer Ignition...
DEBUG RendezvousIP from the AgentConfig 10.40.1.55
INFO The rendezvous host IP (node0 IP) is 10.40.1.55
DEBUG Generated random infra-env id 1912d5c5-feb8-4035-8dd5-0a74e2f64c6b
DEBUG Fetching BaseIso Image...
DEBUG Fetching Agent Manifests...
DEBUG Reusing previously-fetched Agent Manifests
DEBUG Fetching Install Config...
DEBUG Reusing previously-fetched Install Config
DEBUG Fetching Mirror Registries Config...
DEBUG Reusing previously-fetched Mirror Registries Config
DEBUG Generating BaseIso Image...
INFO Extracting base ISO from release payload
DEBUG Using mirror configuration
DEBUG Fetching image from OCP release (oc adm release info --image-for=machine-os-images --insecure=true --icsp-file=/tmp/icsp-file2798914892 quay.io/openshift-release-dev/ocp-release@sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b)
DEBUG extracting /coreos/coreos-x86_64.iso to /root/.cache/agent/image_cache, oc image extract --path /coreos/coreos-x86_64.iso:/root/.cache/agent/image_cache --confirm --icsp-file=/tmp/icsp-file3074929307 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9d36b729016727cb4acf85f858ed437671747b3d3df0afca02e52d2f68ce19
WARNING Failed to extract base ISO from release payload - check registry configuration
INFO Downloading base ISO
DEBUG Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.12/builds/412.86.202301311551-0/x86_64/rhcos-412.86.202301311551-0-live.x86_64.iso'------------------------------------------------------------------------------->download iso from the internet.
DEBUG The file was found in cache: /root/.cache/agent/image_cache/rhcos-412.86.202301311551-0-live.x86_64.iso
DEBUG Using base ISO image /root/.cache/agent/image_cache/rhcos-412.86.202301311551-0-live.x86_64.iso
DEBUG Generating Agent Installer ISO...
DEBUG initDisk(): start
DEBUG initDisk(): regular file
DEBUG trying fat32
DEBUG fat32 failed: Error reading MS-DOS Boot Sector: Could not read FAT32 BIOS Parameter Block from boot sector: Could not read embedded DOS 3.31 BPB: Error reading embedded DOS 2.0 BPB: Invalid sector size 37008 provided in DOS 2.0 BPB. Must be 512
DEBUG trying iso9660 with physical block size 0
INFO Consuming Agent Config from target directory
DEBUG Purging asset "Agent Config" from disk
INFO Consuming Install Config from target directory
DEBUG Purging asset "Install Config" from disk
DEBUG Fetching Kubeconfig Admin Client...
DEBUG Loading Kubeconfig Admin Client...
DEBUG Loading Certificate (admin-kubeconfig-client)...
DEBUG Loading Certificate (admin-kubeconfig-signer)...
DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)...
DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)...
DEBUG Loading Certificate (kube-apiserver-localhost-signer)...
DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)...
DEBUG Loading Certificate (kube-apiserver-service-network-signer)...
DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)...
DEBUG Loading Certificate (kube-apiserver-lb-signer)...
DEBUG Loading ClusterDeployment Config...
DEBUG Fetching Certificate (admin-kubeconfig-client)...
DEBUG Fetching Certificate (admin-kubeconfig-signer)...
DEBUG Reusing previously-fetched Certificate (admin-kubeconfig-signer)
DEBUG Generating Certificate (admin-kubeconfig-client)...
DEBUG Fetching Certificate (kube-apiserver-complete-server-ca-bundle)...
DEBUG Fetching Certificate (kube-apiserver-localhost-ca-bundle)...
DEBUG Fetching Certificate (kube-apiserver-localhost-signer)...
DEBUG Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)
DEBUG Generating Certificate (kube-apiserver-localhost-ca-bundle)...
DEBUG Fetching Certificate (kube-apiserver-service-network-ca-bundle)...
DEBUG Fetching Certificate (kube-apiserver-service-network-signer)...
DEBUG Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)
DEBUG Generating Certificate (kube-apiserver-service-network-ca-bundle)...
DEBUG Fetching Certificate (kube-apiserver-lb-ca-bundle)...
DEBUG Fetching Certificate (kube-apiserver-lb-signer)...
DEBUG Reusing previously-fetched Certificate (kube-apiserver-lb-signer)
DEBUG Generating Certificate (kube-apiserver-lb-ca-bundle)...
DEBUG Generating Certificate (kube-apiserver-complete-server-ca-bundle)...
DEBUG Fetching ClusterDeployment Config...
DEBUG Reusing previously-fetched ClusterDeployment Config
DEBUG Generating Kubeconfig Admin Client...
DEBUG Fetching Kubeadmin Password...
DEBUG Reusing previously-fetched Kubeadmin Password
[root@bastion ~]#
-------------------------------
Here we can see the https://rhcos.mirror.openshift.com is used to grab the ISO. Whereas the logs shows failure with the payload as,
---------------
DEBUG Reusing previously-fetched Release Image Pull Spec
DEBUG Generating Mirror Registries Config...
WARNING The ImageContentSources configuration in install-config.yaml should have at-least one source field matching the releaseImage value quay.io/openshift-release-dev/ocp-release@sha256
DEBUG Fetching Mirror Registries Certificate File...
DEBUG Fetching Install Config...
---------------
At this stage in install-config.yaml we have,
---------------
imageContentSources:
- mirrors:
- ocpregistry.xyz.xz.com:5000/ocp412/openshift/release
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
- ocpregistry.xyz.xz.com:5000/ocp412/openshift/release-images
source: quay.io/openshift-release-dev/ocp-release
---------------
So likely the "openshift-install" isn't pointing to the correct payload as it seems and then gets directed to the mirrored site rather registry.
The registry used is created with oc-mirror plugin from documetation,
https://docs.openshift.com/container-platform/4.12/installing/disconnected_install/installing-mirroring-disconnected.html#installing-mirroring-disconnected
which do not specify something like "oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" which I believe marks the correct payload to the openshift-install tool as per the other way to create registry ( https://docs.openshift.com/container-platform/4.12/installing/disconnected_install/installing-mirroring-installation-images.html#installing-mirroring-installation-images )
Note: oc-mirror is used create the registry as starting with version 4.11, only the file-based catalog is supported and OLM is need to customer.
Version-Release number of selected component (if applicable):
4.12
How reproducible:
Always
Steps to Reproduce:
1. Create a local mirror registry with oc-mirror 2. Add the "imageContentSources" to install-config.yaml 3. run the command to generate the ISO with debug logs are enabled and it can be seen the ISO is fetched from mirrored site. Note: If the system is not having internet access then the command will hung for ever.
Actual results:
The RHCOS is fetched from mirror site and not from payload.
Expected results:
It shall fetch the ISO from local registry / payload shall match to local registry.
Additional info:
We tried to use "oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" but it failed with an error as unknown manifests so we believe either we are missing on the configuration to export wrt the registry or as oc-mirror documentation doesn't have it mentioned it could be not supported or not work. Though ready to test any alternative suggested.
- relates to
-
OCPBUGS-10810 Add example for oc-mirror to set up ImageContentSources
-
- Closed
-