-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.13.0
-
None
-
Important
-
No
-
False
-
Description of problem:
When creating a s390x cluster on OCP 4.13 or 4.14 using IPI over IBM cloud, irrespective of using a multi release image (i.e. from multi.ocp.releases.ci.openshift.org/) or an s390x specific release image (i.e. from https://s390x.ocp.releases.ci.openshift.org/), the cluster nodes will always fail to get created.
Version-Release number of selected component (if applicable):
4.13.13, 4.14.0-rc.2
Should be reproducible for any other OCP versions also.
How reproducible:
Always
Steps to Reproduce:
1. Get an openshift-install binary from either CI release controller (multi/s390x) and any 4.13/4.14 release image from it 2. Run openshift-install (on local) by selecting IBM cloud, create an install-config 3. At install-config edit step, select architecture for both control-plane and worker pools as s390x (additionally, select IBM cloud VSI machine type as s390x instance profile like bz2-4x16 or bz2-8x32 or similar) 4. Create manifests, create cluster -- bootstrap and master node creation both will fail.
Actual results:
DEBUG module.image.ibm_cos_bucket_object.file: Creation complete after 16m56s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/3cdf7bdcbafa4dcc944eb3bdd3f8bef5:62b69aad-6365-4ff1-8141-bdfd946318ef:bucket:swghosh-s390x-2023092-jdhfw-vsi-image:object:rhcos-413.92.202307260246-0-ibmcloud.s390x.qcow2:location:eu-de]
DEBUG module.image.ibm_is_image.image: Creating...
DEBUG module.image.ibm_is_image.image: Still creating... [10s elapsed]
---
DEBUG ibm_cos_bucket_object.bootstrap_ignition: Creation complete after 2s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/3cdf7bdcbafa4dcc944eb3bdd3f8bef5:62b69aad-6365-4ff1-8141-bdfd946318ef:bucket:swghosh-s390x-2023092-jdhfw-bootstrap-ignition:object:bootstrap.ign:location:eu-de]
DEBUG ibm_is_instance.bootstrap_node: Creating...
DEBUG ibm_is_security_group_rule.bootstrap_ssh_inbound[0]: Creation complete after 3s [id=r010-b745eb77-3b2d-446f-824b-1997da9461d7.r010-9ec7d888-c84f-4bf3-adbc-4f9870d0c77d]
ERROR
ERROR Error: Image OS architecture amd64 is not supported by the instance profile bz2-4x16
ERROR
ERROR with ibm_is_instance.bootstrap_node,
ERROR on main.tf line 30, in resource "ibm_is_instance" "bootstrap_node":
ERROR 30: resource "ibm_is_instance" "bootstrap_node" {
ERROR
ERROR failed to fetch Cluster: failed to generate asset "Cluster": failure applying terraform for "bootstrap" stage: failed to create cluster: failed to apply Terraform: exit status 1
ERROR
ERROR Error: Image OS architecture amd64 is not supported by the instance profile bz2-4x16
ERROR
ERROR with ibm_is_instance.bootstrap_node,
ERROR on main.tf line 30, in resource "ibm_is_instance" "bootstrap_node":
ERROR 30: resource "ibm_is_instance" "bootstrap_node" {
ERROR
ERROR
Expected results:
s390x nodes should come up and OCP installation should proceed as normal and bootstrap should complete
Additional info:
The RHCOS image for s390x was successfully uploaded to IBM cloud and a custom boot image was created out of it. The image that got used for my specific version, details underneath.
» cat release.txt | head -20 Client tools for OpenShift --------------------------These archives contain the client tooling for [OpenShift](https://docs.openshift.com).To verify the contents of this directory, use the 'gpg' and 'shasum' tools to ensure the archives you have downloaded match those published from this location.The openshift-install binary has been preconfigured to install the following release:---Name: 4.13.13 Digest: sha256:eac141144d2ecd6cf27d24efe9209358ba516da22becc5f0abc199d25a9cfcec Created: 2023-09-14T09:13:44Z OS/Arch: multi (linux/amd64) Manifests: 658 Metadata files: 1Pull From: quay.io/openshift-release-dev/ocp-release@sha256:eac141144d2ecd6cf27d24efe9209358ba516da22becc5f0abc199d25a9cfcec » ./openshift-install coreos print-stream-json | grep ibmcloud -A 25 | grep -B 10 s390x "ibmcloud": { "release": "413.92.202307260246-0", "formats": { "qcow2.gz": { "disk": { "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.13-9.2/builds/413.92.202307260246-0/s390x/rhcos-413.92.202307260246-0-ibmcloud.s390x.qcow2.gz", -- "uncompressed-sha256": "ea907c9ea1d4dc6780d1233ffcd0bbaec50e9ae767f4de8a468f9618e04114fc" } } } }, "metal": { "release": "413.92.202307260246-0", "formats": { "4k.raw.gz": { "disk": { "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.13-9.2/builds/413.92.202307260246-0/s390x/rhcos-413.92.202307260246-0-metal4k.s390x.raw.gz", "sha256": "074285cdcfb065f6671f6972cd27eb6a6ee6b9031e8d5f9f564a199a643649ed", "uncompressed-sha256": "b6ed18dac25ed7bac5e7d6a0e478180999e422241af3522c62a311ed7c718cc6" } }, "iso": { "disk": { "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.13-9.2/builds/413.92.202307260246-0/s390x/rhcos-413.92.202307260246-0-live.s390x.iso", "sha256": "604b30ed3a9506933db14f3a91138742e91e69516e1fb6d6727fccf746417f82" } }, "pxe": { "kernel": { "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.13-9.2/builds/413.92.202307260246-0/s390x/rhcos-413.92.202307260246-0-live-kernel-s390x",
The issue appears to be that the RHCOS image after getting into IBM cloud as a custom boot image has OS selected as "Red Hat Enterprise Linux" by the installer and IBM cloud's API calls probably don't support specifying an image architecture when the custom image creation occurs. This algo-magically would let IBM cloud infer that it is an x86_64 arch OS image, and hence during creation of s390x virtual server instance attempts, openshift installer would error out naturally (though correct s390x.qcow2 image (and not amd64) was uploaded during install time).
