[3459289953] Upstream Reporter: Aashish Radhakrishnan
Upstream issue status: Closed
Upstream description:

The c9s & c10s [builds](https://redhat-internal.slack.com/archives/C04668B0XK2/p1758814291304339) for the s390x architecture is failing the luks.cex test with Segmentation violation.

```
2025-09-25T15:31:02.982Z] — FAIL: luks.cex (41.29s)
[2025-09-25T15:31:02.982Z] panic: runtime error: invalid memory address or nil pointer dereference [recovered]
[2025-09-25T15:31:02.982Z] panic: runtime error: invalid memory address or nil pointer dereference
[2025-09-25T15:31:02.983Z] [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7572ca]
[2025-09-25T15:31:02.983Z]
[2025-09-25T15:31:02.983Z] goroutine 144 [running]:
[2025-09-25T15:31:02.983Z] github.com/coreos/coreos-assembler/mantle/harness.tRunner.func1()
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/harness/harness.go:519 +0x350
[2025-09-25T15:31:02.983Z] panic(

Unknown macro: {0x24e0b00, 0x45403b0}

)
[2025-09-25T15:31:02.983Z] /usr/lib/golang/src/runtime/panic.go:792 +0x11a
[2025-09-25T15:31:02.983Z] github.com/coreos/coreos-assembler/mantle/platform.InstallFile(

Unknown macro: {0x30966a0, 0xc00008a090}

,

Unknown macro: {0x0, 0x0}

,

Unknown macro: {0xc000500140, 0x1c}

)
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/platform/platform.go:299 +0xca
[2025-09-25T15:31:02.983Z] github.com/coreos/coreos-assembler/mantle/kola.ScpKolet(

Unknown macro: {0xc000727a80, 0x1, 0x1}

)
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/kola/harness.go:1974 +0x528
[2025-09-25T15:31:02.983Z] github.com/coreos/coreos-assembler/mantle/kola/tests/ignition.runCexTest({0xc0000f9e00,

Unknown macro: {0x30be498, 0xc0008be040}

,

Unknown macro: {0xc000aa4090, 0x1, 0x1}

, 0x0, 0x0})
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/kola/tests/ignition/luks.go:250 +0x150
[2025-09-25T15:31:02.983Z] github.com/coreos/coreos-assembler/mantle/kola.runTest(0xc0000f9e00, 0xc0005c7d48,

Unknown macro: {0x3ffc3378c54, 0x4}

,

Unknown macro: {0x30bb008, 0xc000544030}

)
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/kola/harness.go:1942 +0xfa2
[2025-09-25T15:31:02.983Z] github.com/coreos/coreos-assembler/mantle/kola.runProvidedTests.func1(0xc0000f9e00)
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/kola/harness.go:847 +0x88
[2025-09-25T15:31:02.983Z] github.com/coreos/coreos-assembler/mantle/harness.tRunner(0xc0000f9e00, 0xc000b7af30)
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/harness/harness.go:561 +0x170
[2025-09-25T15:31:02.983Z] created by github.com/coreos/coreos-assembler/mantle/harness.(*H).RunTimeout in goroutine 1
[2025-09-25T15:31:02.983Z] /root/containerbuild/mantle/harness/harness.go:607 +0x42c
[2025-09-25T15:31:02.983Z] failed to execute cmd-kola: exit status 2
```

The boot process drops to an emergency shell after ignition-ostree-growfs.service exits with a failure.

The log for this service explicitly states `Failed to open key file`. The service needs to unlock the LUKS device to resize the filesystem within but it cannot access the key located at `/etc/luks/cex.key` from within the initramfs environment.

```
[ 31.174472] ignition-ostree-growfs[1128]: CHANGED: partition=4 start=788480 old: size=6557696 end=7346175 new: size=32765919 end=33554398
[ 31.286100] ignition-ostree-growfs[1242]: null
[ 31.291173] ignition-ostree-growfs[1243]: true
[ 31.297500] ignition-ostree-growfs[1244]: Failed to open key file.

```

From the initial investigation, it looks like the `ignition-ostree-growfs` service (in the initramfs) cannot access the LUKS key required to unlock and resize the encrypted root filesystem even though the CEX hardware is passed through to the VM.

May be we need to figure out if we can equate [coreos-s90x-builder.bu](https://github.com/coreos/fedora-coreos-pipeline/blob/main/multi-arch-builders/coreos-s390x-builder.bu) to [coreos-c390x-rhcos-builder.bu](https://github.com/coreos/fedora-coreos-pipeline/blob/main/multi-arch-builders/coreos-s390x-rhcos-builder.bu)

[console.txt](https://github.com/user-attachments/files/22568779/console.txt)