-
Bug
-
Resolution: Done
-
Major
-
None
-
4.12
-
None
-
Important
-
None
-
Sprint 228, Multi-Arch Sprint 229, Multi-Arch Sprint 230, Multi-Arch Sprint 231, Multi-Arch Sprint 232
-
5
-
Rejected
-
False
-
Description of problem:
Starting with OCP 4.12.0-ec.4 on Z build and continuing through the latest OCP 4.12.0.0 nightly build on Z, 4.12.0-0.nightly-s390x-2022-10-16-225440, all zVM installs fail when using the FIPS=TRUE installation option.
Version-Release number of selected component (if applicable):
OCP 4.12.0-ec.4 on Z build through the latest OCP 4.12.0.0 nightly build on Z, 4.12.0-0.nightly-s390x-2022-10-16-225440.
How reproducible:
Consistently reproducible when specifying the FIPS=TRUE installation option.
Steps to Reproduce:
1.Configure the install-config.yaml file with the FIPS=TRUE option. 2.Proceed with a standard connected installation.
Actual results:
The bootstrap node boots, but none of the master nodes complete their boot. Please see the bootstrap node's "journalctl -b -f -u release-image.service -u bootkube.service" output below.
Expected results:
The master nodes should all complete their boots and the installation successfully proceed and complete, as it does when the FIPS=FALSE installation option is specified.
Additional info:
1. When the FIPS=FALSE option is specified with any of the affected OCP 4.12.0-ec.4 and post nightly OCP 4.12.0 nightly builds, all of the installations complete (forgoing any other issues). 2. The OCP 4.12.0-ec.0 through OCP 4.12.0.ec.3 builds all successfully install with the FIPS=TRUE option, for both KVM and zVM environments. 3. Here is a snippet of the bootstrap node's "journalctl -b -f -u release-image.service -u bootkube.service" command for the OCP 4.12.0-0.nightly-s390x-2022-10-16-225440 build when FIPS=TRUE and the installation fails: Oct 17 03:43:29 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[14925]: WARNING: Validity period of the certificate for "etcd-signer" is greater than 5 years! Oct 17 03:43:29 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[14925]: WARNING: By security reasons it is strongly recommended to change this period an d make it smaller! Oct 17 03:43:31 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[14970]: F1017 03:43:31.641959 1 render.go:61] tls: failed to parse private key Oct 17 03:43:31 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[14925]: F1017 03:43:31.641959 1 render.go:61] tls: failed to parse private key Oct 17 03:43:32 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Main process exited, code=exited, status=255/n/a Oct 17 03:43:32 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Failed with result 'exit-code'. Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Service RestartSec=5s expired, scheduling restart. Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 6. Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Stopped Bootstrap a Kubernetes cluster. Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Started Bootstrap a Kubernetes cluster. Oct 17 03:43:38 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15315]: Oct 17 03:43:39 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com infallible_cray[15339]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e6189dc450b38665510da8 983dbd272daa9975325fb79ce0a3a8b16bd6d4ed44 Oct 17 03:43:39 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15411]: Oct 17 03:43:40 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com keen_lovelace[15431]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aea93282ea25f019d16abf4e 27b157d3b602bebf7ac129c349bca382ef9edaab Oct 17 03:43:41 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15526]: Oct 17 03:43:42 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com epic_maxwell[15551]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fb7d617a8392651c39a47e808ee23f647b1c4cd76cd4bab2304d7185cc0ceb11 Oct 17 03:43:42 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15627]: Oct 17 03:43:43 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com beautiful_mahavira[15643]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:861a24b17366455b98f82efaf561a9dcc0cbfec6e4a4155478b1ea97556f1efa Oct 17 03:43:44 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15716]: Oct 17 03:43:45 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com vigilant_margulis[15735]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c5ecab014ddf893bfdf469c1858e24c88f2b4427caf4a4017b055846b66b6f0 Oct 17 03:43:45 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15805]: Oct 17 03:43:46 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com kind_bell[15830]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0272401995ea57c79bbee2f01caf38d6af0da37effbc18a817fd9335e4cceead Oct 17 03:43:47 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15898]: Oct 17 03:43:48 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com sharp_ramanujan[15918]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75406888a967e91066f8dcae8beefbdaf52808797064d33aa05bcdba0d0f7354 Oct 17 03:43:48 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15989]: Oct 17 03:43:49 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com dazzling_mcnulty[16009]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6fe2a3575866b9e34977efdaf4d36a7a16ed79ab5d8fe8f333762f4ae9fdec1 Oct 17 03:43:50 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16081]: Oct 17 03:43:51 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com sweet_sanderson[16104]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c066fdb2c7700c3e77c1fa5b3b2a515c09a5df4c15d849c08eb73aff725a9a13 Oct 17 03:43:51 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16171]: Oct 17 03:43:52 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com lucid_lederberg[16197]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:857585b4966aa048fcbe04ff0efacdbe63676e847fd876969de87a4abc3dfafe Oct 17 03:43:53 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16272]: Oct 17 03:43:54 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com elastic_lederberg[16286]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e4a43d36edd7fb572d922b7a635952e63e7aff6c3b7bdc3c43dd15d786a1c24 Oct 17 03:43:54 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16352]: Oct 17 03:43:55 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com magical_wiles[16372]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33681baa5c82c3804de29d3e5d3bebd9b72286ba6fb1d655056a1bb01f5d6ae0 Oct 17 03:43:56 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16445]: Oct 17 03:43:56 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com vibrant_sinoussi[16467]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0672808221de4fd020c360b325e4616914ea9f871aa6a9da8a4df6f554408046 Oct 17 03:43:57 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16539]: Oct 17 03:43:58 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com brave_dijkstra[16561]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:329b412afd08f48708c9009591fedaba36b675ca6e2fb5c235a652db4e5632dd Oct 17 03:43:59 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16635]: Oct 17 03:44:00 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com zealous_maxwell[16657]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90bc20e486864e987f7751874e8bb69552e1b7e9665917bbd1fc232a253c4ba6 Oct 17 03:44:00 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16732]: Oct 17 03:44:01 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com strange_joliot[16778]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0e6aeaaf1bf4826772ad55c8a56541d3b6929de95deb163c6672a26b60d360b Oct 17 03:44:02 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16859]: Oct 17 03:44:02 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com musing_keldysh[16884]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8450f786670b01b8670f9a0424c8a2da38e8fac1791d8defa63995966837f135 Oct 17 03:44:03 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16952]: Oct 17 03:44:04 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com youthful_cannon[16971]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42047c76ac50fb1b9ab1d8011fa91cecfca80b3b4b5c7ba5ac8020e64142d6b8 Oct 17 03:44:04 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[15268]: Rendering CEO Manifests... Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[17049]: Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970687 1 bootstrap_ip_linux.go:35] retrieved Address map map[0xc00003d110:[127.0.0.1/8 lo ::1/128] 0xc00003d1e0:[10.20.116.10/24 enc2e0]] Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970782 1 bootstrap_ip_linux.go:54] Ignoring route non Router advertisement route {Ifindex: 1 Dst: ::1/128 Src: <nil> Gw: <nil> Flags: [] Table: 254} Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970687 1 bootstrap_ip_linux.go:35] retrieved Address map map[0xc00003d110:[127.0.0.1/8 lo ::1/128] 0xc00003d1e0:[10.20.116.10/24 enc2e0]] Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970782 1 bootstrap_ip_linux.go:54] Ignoring route non Router advertisement route {Ifindex: 1 Dst: ::1/128 Src: <nil> Gw: <nil> Flags: [] Table: 254} Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970794 1 bootstrap_ip_linux.go:64] Retrieved route map map[] Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970806 1 bootstrap_ip.go:158] Filtered address 127.0.0.1/8 lo Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970812 1 bootstrap_ip.go:158] Filtered address ::1/128 Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970818 1 bootstrap_ip.go:158] Filtered address 10.20.116.10/24 enc2e0 Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970821 1 bootstrap_ip.go:200] Found routable IPs [] Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: W1017 03:44:05.970827 1 bootstrap_ip.go:82] couldn't detect the bootstrap IP automatically, falling back to the first listed address Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970831 1 render.go:414] using bootstrap IP 10.20.116.10 Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970838 1 render.go:597] Bootstrapping etcd using: "HAScalingStrategy" Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: WARNING: Validity period of the certificate for "etcd-signer" is greater than 5 years! Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: WARNING: By security reasons it is strongly recommended to change this period and make it smaller! Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970794 1 bootstrap_ip_linux.go:64] Retrieved route map map[] Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970806 1 bootstrap_ip.go:158] Filtered address 127.0.0.1/8 lo Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970812 1 bootstrap_ip.go:158] Filtered address ::1/128 Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970818 1 bootstrap_ip.go:158] Filtered address 10.20.116.10/24 enc2e0 Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970821 1 bootstrap_ip.go:200] Found routable IPs [] Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: W1017 03:44:05.970827 1 bootstrap_ip.go:82] couldn't detect the bootstrap IP automatically, falling back to the first listed address Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970831 1 render.go:414] using bootstrap IP 10.20.116.10 Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970838 1 render.go:597] Bootstrapping etcd using: "HAScalingStrategy" Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: WARNING: Validity period of the certificate for "etcd-signer" is greater than 5 years! Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: WARNING: By security reasons it is strongly recommended to change this period and make it smaller! Oct 17 03:44:06 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: F1017 03:44:06.619069 1 render.go:61] tls: failed to parse private key Oct 17 03:44:06 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: F1017 03:44:06.619069 1 render.go:61] tls: failed to parse private key Oct 17 03:44:07 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Main process exited, code=exited, status=255/n/a Oct 17 03:44:07 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Failed with result 'exit-code'. Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Service RestartSec=5s expired, scheduling restart. Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 7. Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Stopped Bootstrap a Kubernetes cluster. Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Started Bootstrap a Kubernetes cluster. Oct 17 03:44:13 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[17210]: 3.
- account is impacted by
-
OCPBUGS-4327 Follow-up for OCCPBUGS-2442: FIPS must-gather information
- Closed
- is blocked by
-
OCPBUGS-2396 FIPS jobs are broken after images rebuilt with golang 1.19
- Closed
-
OCPBUGS-6502 Update container node rpms to pick up golang fixes
- Closed