[OCPBUGS-33651] oc command cannot be used with RHEL 8 based bastion on s390x

Type: Bug
Resolution: Done-Errata
Priority: Critical
Fix Version/s: None
Affects Version/s: 4.16.0
Component/s: oc
Labels:
None

Severity:
Critical
Regression:
Yes
Release Blocker:
Rejected
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Release Note Text:
N/A
Release Note Type:
Release Note Not Required
Release Note Status:
Done
Target Version:

4.17.0
Target Backport Versions:

4.16.0

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:

oc command cannot be used with RHEL 8 based bastion

Version-Release number of selected component (if applicable):

    4.16.0-rc.1

How reproducible:

    Very

Steps to Reproduce:

    1. Have a bastion for z/VM installation at Red Hat Enterprise Linux release 8.9 (Ootpa) 
    2. Download and install the 4.16.0-rc.1 client on the bastion
    3.Attempt to use the oc command

Actual results:

    oc get nodes
oc: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by oc)
oc: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by oc)
oc: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by oc)

Expected results:

    oc command returns without error

Additional info:

    This was introduced in 4.16.0-rc.1 - 4.16.0-rc.0 works fine

blocks

OCPBUGS-33916 oc command cannot be used with RHEL 8 based bastion on s390x

Closed

is cloned by

OCPBUGS-33916 oc command cannot be used with RHEL 8 based bastion on s390x

Closed

links to

openshift/oc#1766: OCPBUGS-33651: Add s390x based oc into extraction target

RHEA-2024:3718 OpenShift Container Platform 4.17.z bug fix update

Errata Tool added a comment - 2024/10/01 5:31 PM

Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

For information on the advisory (Moderate: OpenShift Container Platform 4.17.0 bug fix and security update), and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:3718

Errata Tool added a comment - 2024/10/01 5:31 PM Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Moderate: OpenShift Container Platform 4.17.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:3718

Christian LaPolt (Inactive) added a comment - 2024/06/04 7:35 PM

I have run through several installs and upgrades with 4.16.0-rc.3 without issue using the RHEL8 client included in the payload. If we all agree, I feel this can be closed.

Christian LaPolt (Inactive) added a comment - 2024/06/04 7:35 PM I have run through several installs and upgrades with 4.16.0-rc.3 without issue using the RHEL8 client included in the payload. If we all agree, I feel this can be closed.

Christian LaPolt (Inactive) added a comment - 2024/06/01 3:36 AM

4.16.0-rc.3 is out on the mirror site. There are separate client tar files there, 1 for RHEL8 and 1 for RHEL 9.
Client - https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.3/openshift-client-linux-s390x-rhel8-4.16.0-rc.3.tar.gz
Client - https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.3/openshift-client-linux-s390x-rhel9-4.16.0-rc.3.tar.gz
Installer - https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.3/openshift-install-linux.tar.gz

The RHEL8 z/VM install went fine.

Christian LaPolt (Inactive) added a comment - 2024/06/01 3:36 AM 4.16.0-rc.3 is out on the mirror site. There are separate client tar files there, 1 for RHEL8 and 1 for RHEL 9. Client - https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.3/openshift-client-linux-s390x-rhel8-4.16.0-rc.3.tar.gz Client - https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.3/openshift-client-linux-s390x-rhel9-4.16.0-rc.3.tar.gz Installer - https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.3/openshift-install-linux.tar.gz The RHEL8 z/VM install went fine.

Ying Zhou added a comment - 2024/05/22 9:48 AM

tested with latest accepted payload, the fixed works as expected :

./oc adm release extract  --command-os='linux/s390x'   --tools registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-21-233006  --to=/tmp/octest

./oc adm release extract  --command-os='linux/s390x' --command='oc.rhel8'  registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-21-233006  --to=/tmp/octest

ll octest/
total 237916
-rwxr-xr-x. 1 root root 132997430  5月22日 02:56 oc
-rwxr-xr-x. 1 root root  55290711  5月22日 02:56 openshift-client-linux-s390x-rhel8-4.17.0-0.nightly-2024-05-21-233006.tar.gz
-rwxr-xr-x. 1 root root  55288433  5月22日 03:05 openshift-client-linux-s390x-rhel9-4.17.0-0.nightly-2024-05-21-233006.tar.gz
-rw-r--r--. 1 root root     33066  5月22日 17:43 release.txt
-rw-r--r--. 1 root root       364  5月22日 17:43 sha256sum.txt

Ying Zhou added a comment - 2024/05/22 9:48 AM tested with latest accepted payload, the fixed works as expected : ./oc adm release extract --command-os= 'linux/s390x' --tools registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-21-233006 --to=/tmp/octest ./oc adm release extract --command-os= 'linux/s390x' --command= 'oc.rhel8' registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-21-233006 --to=/tmp/octest ll octest/ total 237916 -rwxr-xr-x. 1 root root 132997430 5月22日 02:56 oc -rwxr-xr-x. 1 root root 55290711 5月22日 02:56 openshift-client-linux-s390x-rhel8-4.17.0-0.nightly-2024-05-21-233006.tar.gz -rwxr-xr-x. 1 root root 55288433 5月22日 03:05 openshift-client-linux-s390x-rhel9-4.17.0-0.nightly-2024-05-21-233006.tar.gz -rw-r--r--. 1 root root 33066 5月22日 17:43 release.txt -rw-r--r--. 1 root root 364 5月22日 17:43 sha256sum.txt

Ying Zhou added a comment - 2024/05/20 9:04 AM

I could extract from the payload :

./oc adm release extract  --command-os='linux/s390x' --command='oc.rhel8' registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-18-152118  --to=/tmp/ 

./oc adm release extract  --command-os='linux/s390x'   --tools  registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-18-152118  --to=/tmp/

ll /tmp
total 108044
-rwxr-xr-x. 1 root root 55297774  5月18日 16:59 openshift-client-linux-s390x-rhel8-4.17.0-0.nightly-2024-05-18-152118.tar.gz
-rwxr-xr-x. 1 root root 55295900  5月18日 17:06 openshift-client-linux-s390x-rhel9-4.17.0-0.nightly-2024-05-18-152118.tar.gz
-rw-r--r--. 1 root root    33066  5月20日 17:01 release.txt
-rw-r--r--. 1 root root      364  5月20日 17:01 sha256sum.txt

But since we don't have accepted payload now, will wait more time.

Ying Zhou added a comment - 2024/05/20 9:04 AM I could extract from the payload : ./oc adm release extract --command-os= 'linux/s390x' --command= 'oc.rhel8' registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-18-152118 --to=/tmp/ ./oc adm release extract --command-os= 'linux/s390x' --tools registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-05-18-152118 --to=/tmp/ ll /tmp total 108044 -rwxr-xr-x. 1 root root 55297774 5月18日 16:59 openshift-client-linux-s390x-rhel8-4.17.0-0.nightly-2024-05-18-152118.tar.gz -rwxr-xr-x. 1 root root 55295900 5月18日 17:06 openshift-client-linux-s390x-rhel9-4.17.0-0.nightly-2024-05-18-152118.tar.gz -rw-r--r--. 1 root root 33066 5月20日 17:01 release.txt -rw-r--r--. 1 root root 364 5月20日 17:01 sha256sum.txt But since we don't have accepted payload now, will wait more time.

OpenShift Jira Bot added a comment - 2024/05/18 7:24 AM

Hi aguclu@redhat.com,

Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

OpenShift Jira Bot added a comment - 2024/05/18 7:24 AM Hi aguclu@redhat.com , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

Arda Guclu added a comment - 2024/05/15 1:59 PM

Thanks for the detailed explanation.

> https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.0/openshift-client-linux.tar.gz
We collect each of the non-amd64 arches into their own arch-specific bucket representing the target architecture of the cluster you want to install on. Within that bucket, the listing of the binaries that don't contain an architecture matches the architecture of the target cluster as specificied in the URL.

We are not populating these arch specific urls (openshift-v4/s390x, openshift-v4/arm64, etc.). Probably this is a better question to ART team (maybe?, or they can redirect correct team). But in any case, I assume that arch name will exist in the url name (apart from the default one), because they are all generated from the same command (e.g. oc adm release extract --tools)

> To contrast, the installer has already provided statically linked options for both RHEL 8 and RHEL 9:

I checked https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.1/ and don't see rhel8 based installer in that list. I think, this question also goes to ART.

I repeated this multiple times but best thing is to switch to RHEL9. If it is not possible, in order to unblock you, we can start adding s390x based oc into all the archs (openshift-client-linux-s390x-rhel8-4.16.0-rc.1.tar.gz). Unfortunately we can't change the naming, it should be compatible with the other names. Perhaps, there is a chance that whoever populates this url https://mirror.openshift.com/pub/openshift-v4/s390x/ also renames according to your needs.

Arda Guclu added a comment - 2024/05/15 1:59 PM Thanks for the detailed explanation. > https://mirror.openshift.com/pub/openshift-v4/ s390x /clients/ocp/4.16.0-rc.0/openshift-client-linux.tar.gz We collect each of the non-amd64 arches into their own arch-specific bucket representing the target architecture of the cluster you want to install on. Within that bucket, the listing of the binaries that don't contain an architecture matches the architecture of the target cluster as specificied in the URL. We are not populating these arch specific urls (openshift-v4/s390x, openshift-v4/arm64, etc.). Probably this is a better question to ART team (maybe?, or they can redirect correct team). But in any case, I assume that arch name will exist in the url name (apart from the default one), because they are all generated from the same command (e.g. oc adm release extract --tools) > To contrast, the installer has already provided statically linked options for both RHEL 8 and RHEL 9: https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-install-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-install-rhel8-s390x.tar.gz I checked https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.1/ and don't see rhel8 based installer in that list. I think, this question also goes to ART. I repeated this multiple times but best thing is to switch to RHEL9. If it is not possible, in order to unblock you, we can start adding s390x based oc into all the archs (openshift-client-linux-s390x-rhel8-4.16.0-rc.1.tar.gz). Unfortunately we can't change the naming, it should be compatible with the other names. Perhaps, there is a chance that whoever populates this url https://mirror.openshift.com/pub/openshift-v4/s390x/ also renames according to your needs.

Jeremy Poulin added a comment - 2024/05/15 1:34 PM - edited

aguclu@redhat.com The bug report from Christian is specifically about the missing oc.rhel8 binary on RHEL 8 in the mirror URL path that would target installing on an s390x target cluster.

If you notice the URL pattern:
https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.0/openshift-client-linux.tar.gz
We collect each of the non-amd64 arches into their own arch-specific bucket representing the target architecture of the cluster you want to install on. Within that bucket, the listing of the binaries that don't contain an architecture matches the architecture of the target cluster as specificied in the URL.

So in the case of s390x, we would expect both of these to be s390x binaries because the arch is omitted, so it defaults to the target arch in the URL:
https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-client-linux-rhel8.tar.gz
https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-client-linux-rhel9.tar.gz

Instead, only the rhel9 binary is available, and it doesn't specify that it's RHEL 9 specific:
https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-client-linux.tar.gz

To contrast, the installer has already provided statically linked options for both RHEL 8 and RHEL 9:

The first binary should probably specificy rhel9 instead of leaving it ambiguous. (Maybe a symlink would be useful here?)
The name of the second binary is also breaking our naming convention:
I would expect it to be called openshift-install-linux-rhel8.tar.gz (since the s390x part is implied).

All of the above assumes that the binaries live in the client bucket targeting the installation of a s390x cluster.

–
For the amd64 bucket (e.g. https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/), we need to tag the client binaries with both the RHEL version AND the client arch.
For example, here are the power binaries:
openshift-client-linux-ppc64le-rhel8.tar.gz
openshift-client-linux-ppc64le-rhel9.tar.gz

What is weird to me is that we've never published s390x binaries. We probably don't need to, since I don't see an influx of s390x customers wanted to install amd64 clusters from their mainframe anytime soon. That said, for the sake of consistency, I feel like we should copy the binaries their too with the arch-tagged names just to be consistent. I.e.:
openshift-client-linux-s390x-rhel8.tar.gz
openshift-client-linux-s390x-rhel9.tar.gz

Jeremy Poulin added a comment - 2024/05/15 1:34 PM - edited aguclu@redhat.com The bug report from Christian is specifically about the missing oc.rhel8 binary on RHEL 8 in the mirror URL path that would target installing on an s390x target cluster. If you notice the URL pattern: https://mirror.openshift.com/pub/openshift-v4/ s390x /clients/ocp/4.16.0-rc.0/openshift-client-linux.tar.gz We collect each of the non-amd64 arches into their own arch-specific bucket representing the target architecture of the cluster you want to install on. Within that bucket, the listing of the binaries that don't contain an architecture matches the architecture of the target cluster as specificied in the URL. So in the case of s390x, we would expect both of these to be s390x binaries because the arch is omitted, so it defaults to the target arch in the URL: https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-client-linux-rhel8.tar.gz https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-client-linux-rhel9.tar.gz Instead, only the rhel9 binary is available, and it doesn't specify that it's RHEL 9 specific: https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-client-linux.tar.gz To contrast, the installer has already provided statically linked options for both RHEL 8 and RHEL 9: https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-install-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp-dev-preview/latest/openshift-install-rhel8-s390x.tar.gz The first binary should probably specificy rhel9 instead of leaving it ambiguous. (Maybe a symlink would be useful here?) The name of the second binary is also breaking our naming convention: I would expect it to be called openshift-install-linux-rhel8.tar.gz (since the s390x part is implied). All of the above assumes that the binaries live in the client bucket targeting the installation of a s390x cluster. – For the amd64 bucket (e.g. https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/ ), we need to tag the client binaries with both the RHEL version AND the client arch. For example, here are the power binaries: openshift-client-linux-ppc64le-rhel8.tar.gz openshift-client-linux-ppc64le-rhel9.tar.gz What is weird to me is that we've never published s390x binaries. We probably don't need to, since I don't see an influx of s390x customers wanted to install amd64 clusters from their mainframe anytime soon. That said, for the sake of consistency, I feel like we should copy the binaries their too with the arch-tagged names just to be consistent. I.e.: openshift-client-linux-s390x-rhel8.tar.gz openshift-client-linux-s390x-rhel9.tar.gz

Arda Guclu added a comment - 2024/05/15 8:45 AM

christian.lapolt I investigated this issue deeply. From the oc point of view, in order to unblock you, we can start generating oc.rhel8 binary in s390x as we did for the others (we are already compiling it https://github.com/openshift/oc/blob/1073b293db8e3310a51d03294113b660cdde61f0/Makefile#L141-L143 but simply not populating it into the urls above ^) and you can access via something like this `https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.16.0-rc.0/openshift-client-linux-s390x-rhel8.tar.gz`.

However, as far as I can see, this does not fully solve your problem because you still need openshift-installer binary in rhel8 and s390x (it is better to contact with openshift-installer team with a separate issue). And there will be other binaries would block you.

Instead I highly recommend switching your host to RHEL9. If you prefer not to do it, let me know about the oc s390x change and I'll manage opening PR.

Arda Guclu added a comment - 2024/05/15 8:45 AM christian.lapolt I investigated this issue deeply. From the oc point of view, in order to unblock you, we can start generating oc.rhel8 binary in s390x as we did for the others (we are already compiling it https://github.com/openshift/oc/blob/1073b293db8e3310a51d03294113b660cdde61f0/Makefile#L141-L143 but simply not populating it into the urls above ^) and you can access via something like this ` https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.16.0-rc.0/openshift-client-linux-s390x-rhel8.tar.gz `. However, as far as I can see, this does not fully solve your problem because you still need openshift-installer binary in rhel8 and s390x (it is better to contact with openshift-installer team with a separate issue). And there will be other binaries would block you. Instead I highly recommend switching your host to RHEL9. If you prefer not to do it, let me know about the oc s390x change and I'll manage opening PR.

Christian LaPolt (Inactive) added a comment - 2024/05/15 6:57 AM - edited

Here is the list of what was used to attempt the the install.
https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/pre-release/4.16.0-ec.5
Because that is all that is there.
https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.0/openshift-client-linux.tar.gz
https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.0/openshift-install-linux.tar.gz

Once trying rc.1 nothing worked. Never seen this issue before.

https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.1/

Christian LaPolt (Inactive) added a comment - 2024/05/15 6:57 AM - edited Here is the list of what was used to attempt the the install. https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/pre-release/4.16.0-ec.5 Because that is all that is there. https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.0/openshift-client-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.0/openshift-install-linux.tar.gz Once trying rc.1 nothing worked. Never seen this issue before. https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.16.0-rc.1/

Assignee:: Arda Guclu

Reporter:: Christian LaPolt (Inactive)

QA Contact:: Ying Zhou

Contributors:: Christian LaPolt (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Created:: 2024/05/14 3:05 PM

Updated:: 2025/01/09 5:38 PM

Resolved:: 2024/10/01 5:31 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Errata Tool added a comment - 2024/10/01 5:31 PM

Expand comment: Errata Tool added a comment - 2024/10/01 5:31 PM

Collapse comment: Christian LaPolt (Inactive) added a comment - 2024/06/04 7:35 PM

Expand comment: Christian LaPolt (Inactive) added a comment - 2024/06/04 7:35 PM

Collapse comment: Christian LaPolt (Inactive) added a comment - 2024/06/01 3:36 AM

Expand comment: Christian LaPolt (Inactive) added a comment - 2024/06/01 3:36 AM

Collapse comment: Ying Zhou added a comment - 2024/05/22 9:48 AM

Expand comment: Ying Zhou added a comment - 2024/05/22 9:48 AM

Collapse comment: Ying Zhou added a comment - 2024/05/20 9:04 AM

Expand comment: Ying Zhou added a comment - 2024/05/20 9:04 AM

Collapse comment: OpenShift Jira Bot added a comment - 2024/05/18 7:24 AM

Expand comment: OpenShift Jira Bot added a comment - 2024/05/18 7:24 AM

Collapse comment: Arda Guclu added a comment - 2024/05/15 1:59 PM

Expand comment: Arda Guclu added a comment - 2024/05/15 1:59 PM

Collapse comment: Jeremy Poulin added a comment - 2024/05/15 1:34 PM, Edited by Jeremy Poulin - 2024/05/15 1:46 PM

Expand comment: Jeremy Poulin added a comment - 2024/05/15 1:34 PM, Edited by Jeremy Poulin - 2024/05/15 1:46 PM

Collapse comment: Arda Guclu added a comment - 2024/05/15 8:45 AM

Expand comment: Arda Guclu added a comment - 2024/05/15 8:45 AM

Collapse comment: Christian LaPolt (Inactive) added a comment - 2024/05/15 6:57 AM, Edited by Christian LaPolt - 2024/05/15 6:59 AM

Expand comment: Christian LaPolt (Inactive) added a comment - 2024/05/15 6:57 AM, Edited by Christian LaPolt - 2024/05/15 6:59 AM

People

Dates