Major What were you trying to do that didn't work?
When trying to start a Quadlet service from systemctl, the image cannot be pulled for any reason and a timeout shows up.
# systemctl start fedora-minimal.service Job for fedora-minimal.service failed because the control process exited with error code. See "systemctl status fedora-minimal.service" and "journalctl -xeu fedora-minimal.service" for details. [root@nadav-m-rhel-aarch64-0 RH-app-lifecycle-demo]# systemctl status fedora-minimal.service ● fedora-minimal.service - A minimal fedora container Loaded: loaded (/usr/share/containers/systemd/fedora-minimal.container; generated) Active: activating (start) since Mon 2024-02-26 12:25:36 UTC; 2s ago Main PID: 333640 (podman) Tasks: 25 (limit: 1024707) Memory: 34.9M CPU: 194ms CGroup: /system.slice/fedora-minimal.service └─333640 /usr/bin/podman run --name=systemd-fedora-minimal --cidfile=/run/fedora-minimal.cid --replace --rm --cgroups=split --sdnotify=conmon -d > Feb 26 12:25:36 nadav-m-rhel-aarch64-0 systemd[1]: Starting A minimal fedora container... Feb 26 12:25:36 nadav-m-rhel-aarch64-0 fedora-minimal[333640]: Trying to pull quay.io/pwallrab/fedora-minimal@sha256:ca5c4e0acb46826907e3e8984bdb493bf34fc0b86> Feb 26 12:25:36 nadav-m-rhel-aarch64-0 fedora-minimal[333640]: time="2024-02-26T12:25:36Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initial> Feb 26 12:25:37 nadav-m-rhel-aarch64-0 fedora-minimal[333640]: time="2024-02-26T12:25:37Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initial> Feb 26 12:25:38 nadav-m-rhel-aarch64-0 fedora-minimal[333640]: time="2024-02-26T12:25:38Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initial>
The service status stays as "activating" forever unless the image is pulled manually with Podman. Once that happens, the service can start.
# podman pull quay.io/pwallrab/fedora-minimal@sha256:ca5c4e0acb46826907e3e8984bdb493bf34fc0b8679579a08e2d03670f622c70
Trying to pull quay.io/pwallrab/fedora-minimal@sha256:ca5c4e0acb46826907e3e8984bdb493bf34fc0b8679579a08e2d03670f622c70...
Getting image source signatures
Copying blob c60a0d12eb6c done
Copying config 1763b57a79 done
Writing manifest to image destination
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
1763b57a7951764912e453d8056fd8f2fd15a43e2b828da5141598f505c70b4a
[root@nadav-m-rhel-aarch64-0 RH-app-lifecycle-demo]# date +%H:%M:%S
12:26:09
[root@nadav-m-rhel-aarch64-0 RH-app-lifecycle-demo]# systemctl status fedora-minimal.service
● fedora-minimal.service - A minimal fedora container
Loaded: loaded (/usr/share/containers/systemd/fedora-minimal.container; generated)
Active: active (running) since Mon 2024-02-26 12:26:15 UTC; 925ms ago
Main PID: 336263 (conmon)
Tasks: 21 (limit: 1024707)
Memory: 23.4M
CPU: 283ms
CGroup: /system.slice/fedora-minimal.service
└─runtime
├─336263 /usr/bin/conmon --api-version 1 -c 3e9877c4fcb989d06cb7fbb64e7a638d70654fd0baae37ea058ebb653dd4d0c9 -u 3e9877c4fcb989d06cb7fbb64e7a638>
├─336273 /usr/bin/podman --root /var/lib/containers/storage --runroot /run/containers/storage --log-level warning --cgroup-manager systemd --tm>
├─336292 /usr/libexec/podman/netavark --config /run/containers/networks --rootless=false --aardvark-binary=/usr/libexec/podman/aardvark-dns tea>
└─336334 iptables -t filter -D NETAVARK_FORWARD -d 10.88.0.0/16 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT --wait
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 systemd[1]: Starting A minimal fedora container...
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 fedora-minimal[336174]: WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 podman[336174]: 2024-02-26 12:26:15.330981059 +0000 UTC m=+0.041959703 container create 3e9877c4fcb989d06cb7fbb64e7a638>
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 podman[336174]: 2024-02-26 12:26:15.317205011 +0000 UTC m=+0.028183655 image pull 1763b57a7951764912e453d8056fd8f2fd15a>
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 podman[336174]: 2024-02-26 12:26:15.525449161 +0000 UTC m=+0.236427765 container init 3e9877c4fcb989d06cb7fbb64e7a638d7>
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 systemd[1]: Started A minimal fedora container.
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 systemd-fedora-minimal[336263]: {"msg":"exec container process `/usr/bin/sleep`: Exec format error","level":"error","ti>
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 podman[336174]: 2024-02-26 12:26:15.544534691 +0000 UTC m=+0.255513295 container start 3e9877c4fcb989d06cb7fbb64e7a638d>
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 fedora-minimal[336174]: 3e9877c4fcb989d06cb7fbb64e7a638d70654fd0baae37ea058ebb653dd4d0c9
Feb 26 12:26:15 nadav-m-rhel-aarch64-0 podman[336273]: 2024-02-26 12:26:15.611437006 +0000 UTC m=+0.031085338 container died 3e9877c4fcb989d06cb7fbb64e7a638d7>
Please provide the package NVR for which bug is seen:
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 9.3 (Plow)
$ uname -a
Linux nadav-m-rhel-aarch64-0 5.14.0-284.11.1.el9_2.aarch64 #1 SMP PREEMPT_DYNAMIC Wed Apr 12 11:23:11 EDT 2023 aarch64 aarch64 aarch64 GNU/Linux
$ systemctl --version
systemd 252 (252-18.el9)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
$ podman --version
podman version 4.6.1
Find attached the following files:
- Very detailed sequence of the reproduction scenario: https://issues.redhat.com/secure/attachment/13148028/26Feb2024_ReproductionScenario.log.
- Service configuration file: https://issues.redhat.com/secure/attachment/13148026/26Feb2024_fedora-minimal.container. Log level set to debug.
- Service logs: https://issues.redhat.com/secure/attachment/13148027/26Feb2024_fedora-minimal.service.log.
- SOS Report: https://issues.redhat.com/secure/attachment/13148029/26Feb2024_sosreport-nadav-m-rhel-aarch64-0-AUTOGMSYNC-350-2024-02-26-yhrdqdk.tar.xz.
How reproducible:
The issue could be reproduced until now only from Azureus RHEL instances by our customer. From the machines where I tested this by myself the image is pulled correctly by Systemd.
Steps to reproduce
- Execute systemctl start.
- Wait and confirm that the image cannot be pulled and a timeout shows up.
- Pull the image interactively with Podman.
- Execute systemctl status and confirm that now the image could be pulled and the service is running.
To discard any issue related to SELinux, it was disabled for testing.
Expected results
Systemd can pull the image by itself.
Actual results
The image has to be pulled interactively executing podman pull.
