OCP 4.12 on Z Installations fail when FIPS=TRUE installation option specified for OCP 4.12.0-EC.4 and nightly builds after

Starting with OCP 4.12.0-ec.4 on Z build and continuing through the latest OCP 4.12.0.0 nightly build on Z, 4.12.0-0.nightly-s390x-2022-10-16-225440, all zVM installs fail when using the FIPS=TRUE installation option.  

OCP 4.12.0-ec.4 on Z build through the latest OCP 4.12.0.0 nightly build on Z, 4.12.0-0.nightly-s390x-2022-10-16-225440.

Consistently reproducible when specifying the FIPS=TRUE installation option.

1.Configure the install-config.yaml file with the FIPS=TRUE option. 
2.Proceed with a standard connected installation.

The bootstrap node boots, but none of the master nodes complete their boot.  Please see the bootstrap node's "journalctl -b -f -u release-image.service -u bootkube.service" output below.

The master nodes should all complete their boots and the installation successfully proceed and complete, as it does when the FIPS=FALSE installation option is specified.

1. When the FIPS=FALSE option is specified with any of the affected OCP 4.12.0-ec.4 and post nightly OCP 4.12.0 nightly builds, all of the installations complete (forgoing any other issues).

2. The OCP 4.12.0-ec.0 through OCP 4.12.0.ec.3 builds all successfully install with the FIPS=TRUE option, for both KVM and zVM environments.

3. Here is a snippet of the bootstrap node's "journalctl -b -f -u release-image.service -u bootkube.service" command for the OCP 4.12.0-0.nightly-s390x-2022-10-16-225440 build when FIPS=TRUE and the installation fails:

Oct 17 03:43:29 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[14925]: WARNING: Validity period of the certificate for "etcd-signer" is greater than 5                     years!
Oct 17 03:43:29 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[14925]: WARNING: By security reasons it is strongly recommended to change this period an                    d make it smaller!
Oct 17 03:43:31 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[14970]: F1017 03:43:31.641959       1 render.go:61] tls: failed to parse private key
Oct 17 03:43:31 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[14925]: F1017 03:43:31.641959       1 render.go:61] tls: failed to parse private key
Oct 17 03:43:32 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Main process exited, code=exited, status=255/n/a
Oct 17 03:43:32 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Failed with result 'exit-code'.
Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Service RestartSec=5s expired, scheduling restart.
Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 6.
Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Stopped Bootstrap a Kubernetes cluster.
Oct 17 03:43:37 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Started Bootstrap a Kubernetes cluster.
Oct 17 03:43:38 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15315]:
Oct 17 03:43:39 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com infallible_cray[15339]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e6189dc450b38665510da8                    983dbd272daa9975325fb79ce0a3a8b16bd6d4ed44
Oct 17 03:43:39 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15411]:
Oct 17 03:43:40 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com keen_lovelace[15431]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aea93282ea25f019d16abf4e                    27b157d3b602bebf7ac129c349bca382ef9edaab
Oct 17 03:43:41 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15526]:
Oct 17 03:43:42 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com epic_maxwell[15551]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fb7d617a8392651c39a47e808ee23f647b1c4cd76cd4bab2304d7185cc0ceb11
Oct 17 03:43:42 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15627]:
Oct 17 03:43:43 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com beautiful_mahavira[15643]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:861a24b17366455b98f82efaf561a9dcc0cbfec6e4a4155478b1ea97556f1efa
Oct 17 03:43:44 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15716]:
Oct 17 03:43:45 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com vigilant_margulis[15735]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c5ecab014ddf893bfdf469c1858e24c88f2b4427caf4a4017b055846b66b6f0
Oct 17 03:43:45 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15805]:
Oct 17 03:43:46 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com kind_bell[15830]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0272401995ea57c79bbee2f01caf38d6af0da37effbc18a817fd9335e4cceead
Oct 17 03:43:47 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15898]:
Oct 17 03:43:48 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com sharp_ramanujan[15918]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75406888a967e91066f8dcae8beefbdaf52808797064d33aa05bcdba0d0f7354
Oct 17 03:43:48 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[15989]:
Oct 17 03:43:49 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com dazzling_mcnulty[16009]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6fe2a3575866b9e34977efdaf4d36a7a16ed79ab5d8fe8f333762f4ae9fdec1
Oct 17 03:43:50 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16081]:
Oct 17 03:43:51 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com sweet_sanderson[16104]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c066fdb2c7700c3e77c1fa5b3b2a515c09a5df4c15d849c08eb73aff725a9a13
Oct 17 03:43:51 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16171]:
Oct 17 03:43:52 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com lucid_lederberg[16197]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:857585b4966aa048fcbe04ff0efacdbe63676e847fd876969de87a4abc3dfafe
Oct 17 03:43:53 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16272]:
Oct 17 03:43:54 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com elastic_lederberg[16286]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e4a43d36edd7fb572d922b7a635952e63e7aff6c3b7bdc3c43dd15d786a1c24
Oct 17 03:43:54 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16352]:
Oct 17 03:43:55 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com magical_wiles[16372]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33681baa5c82c3804de29d3e5d3bebd9b72286ba6fb1d655056a1bb01f5d6ae0
Oct 17 03:43:56 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16445]:
Oct 17 03:43:56 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com vibrant_sinoussi[16467]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0672808221de4fd020c360b325e4616914ea9f871aa6a9da8a4df6f554408046
Oct 17 03:43:57 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16539]:
Oct 17 03:43:58 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com brave_dijkstra[16561]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:329b412afd08f48708c9009591fedaba36b675ca6e2fb5c235a652db4e5632dd
Oct 17 03:43:59 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16635]:
Oct 17 03:44:00 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com zealous_maxwell[16657]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90bc20e486864e987f7751874e8bb69552e1b7e9665917bbd1fc232a253c4ba6
Oct 17 03:44:00 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16732]:
Oct 17 03:44:01 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com strange_joliot[16778]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0e6aeaaf1bf4826772ad55c8a56541d3b6929de95deb163c6672a26b60d360b
Oct 17 03:44:02 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16859]:
Oct 17 03:44:02 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com musing_keldysh[16884]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8450f786670b01b8670f9a0424c8a2da38e8fac1791d8defa63995966837f135
Oct 17 03:44:03 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[16952]:
Oct 17 03:44:04 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com youthful_cannon[16971]: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42047c76ac50fb1b9ab1d8011fa91cecfca80b3b4b5c7ba5ac8020e64142d6b8
Oct 17 03:44:04 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[15268]: Rendering CEO Manifests...
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[17049]:
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970687       1 bootstrap_ip_linux.go:35] retrieved Address map map[0xc00003d110:[127.0.0.1/8 lo ::1/128] 0xc00003d1e0:[10.20.116.10/24 enc2e0]]
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970782       1 bootstrap_ip_linux.go:54] Ignoring route non Router advertisement route {Ifindex: 1 Dst: ::1/128 Src: <nil> Gw: <nil> Flags: [] Table: 254}
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970687       1 bootstrap_ip_linux.go:35] retrieved Address map map[0xc00003d110:[127.0.0.1/8 lo ::1/128] 0xc00003d1e0:[10.20.116.10/24 enc2e0]]
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970782       1 bootstrap_ip_linux.go:54] Ignoring route non Router advertisement route {Ifindex: 1 Dst: ::1/128 Src: <nil> Gw: <nil> Flags: [] Table: 254}
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970794       1 bootstrap_ip_linux.go:64] Retrieved route map map[]
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970806       1 bootstrap_ip.go:158] Filtered address 127.0.0.1/8 lo
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970812       1 bootstrap_ip.go:158] Filtered address ::1/128
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970818       1 bootstrap_ip.go:158] Filtered address 10.20.116.10/24 enc2e0
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970821       1 bootstrap_ip.go:200] Found routable IPs []
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: W1017 03:44:05.970827       1 bootstrap_ip.go:82] couldn't detect the bootstrap IP automatically, falling back to the first listed address
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970831       1 render.go:414] using bootstrap IP 10.20.116.10
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: I1017 03:44:05.970838       1 render.go:597] Bootstrapping etcd using: "HAScalingStrategy"
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: WARNING: Validity period of the certificate for "etcd-signer" is greater than 5 years!
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: WARNING: By security reasons it is strongly recommended to change this period and make it smaller!
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970794       1 bootstrap_ip_linux.go:64] Retrieved route map map[]
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970806       1 bootstrap_ip.go:158] Filtered address 127.0.0.1/8 lo
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970812       1 bootstrap_ip.go:158] Filtered address ::1/128
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970818       1 bootstrap_ip.go:158] Filtered address 10.20.116.10/24 enc2e0
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970821       1 bootstrap_ip.go:200] Found routable IPs []
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: W1017 03:44:05.970827       1 bootstrap_ip.go:82] couldn't detect the bootstrap IP automatically, falling back to the first listed address
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970831       1 render.go:414] using bootstrap IP 10.20.116.10
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: I1017 03:44:05.970838       1 render.go:597] Bootstrapping etcd using: "HAScalingStrategy"
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: WARNING: Validity period of the certificate for "etcd-signer" is greater than 5 years!
Oct 17 03:44:05 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: WARNING: By security reasons it is strongly recommended to change this period and make it smaller!
Oct 17 03:44:06 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com etcd-render[17070]: F1017 03:44:06.619069       1 render.go:61] tls: failed to parse private key
Oct 17 03:44:06 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com bootkube.sh[17049]: F1017 03:44:06.619069       1 render.go:61] tls: failed to parse private key
Oct 17 03:44:07 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Main process exited, code=exited, status=255/n/a
Oct 17 03:44:07 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Failed with result 'exit-code'.
Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Service RestartSec=5s expired, scheduling restart.
Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 7.
Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Stopped Bootstrap a Kubernetes cluster.
Oct 17 03:44:12 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com systemd[1]: Started Bootstrap a Kubernetes cluster.
Oct 17 03:44:13 bootstrap-0.pok-20.ocptest.pok.stglabs.ibm.com podman[17210]:




3.