Containers on edpm nodes are created with podman run command and then all other container management is done through systemd files[1].

• Container are being created with podman run -log-driver journald -n <container name>. With PR[1] network agent containers no longer logs to file but container stdout. With this config(-log-driver journald) podman/conmon takes care of writing these stdout to journal, so once container is started we can see the logs with podman logs <container name> or journalctl CONTAINER_NAME=<container name>

• We then Create[2] systemd unit to manage these containers with systemd, As containers are already created in step 1 we do config ExecStart=podman start <container name> and start it and ExecStop=podman stop <container name>. Since these now managed by systemd expectation is as we should see the container logs with systemctl status <service name> or journalctl -u <service name>, but it's not happening until the podman containers or service restart(as then only as systemd starts the pod and add required fieleds like _SYSTEMD_SLICE, _SYSTEMD_CGROUP, _SYSTEMD_UNIT etc)
• Since container is already started podman start is  a noop(set in ExecStart) when systemctl start is triggered in step 2
• When we restart the service or container, the logs get's visible with journalclt -u <service name> or systemctl status <service name>.

Some edpm services like nova_compute do an explicit restart[3] by using ansible restart handlers so the issue got hidden there. Ideally the restart shouldn't be needed on initial deployment as the service is started after all config is done. Also edpm_container_manage takes care of restart with config changes when underlying config is located at /var/lib/config-data/ansible-generated. Not all edpm_services put configuration under this directory so this would not work for every case.

The issue is to track the fix in generalized manner for each services so they correctly log and links to systemd units.

Also another observation is there is a difference in parameters for conmon process when container is stared for the first time and subsequently restarted. This may not be related to the actual issue but an observation but is unknown as of now why this happens so would be good to understand.

--log-level info --exit-command-arg info
vs
--log-level warning --exit-command-arg warning

 [1] https://github.com/openstack-k8s-operators/edpm-ansible/pull/726

 [2] https://github.com/openstack-k8s-operators/edpm-ansible/blob/main/roles/edpm_container_manage/tasks/create.yml#L37-L49

[3] https://github.com/openstack-k8s-operators/edpm-ansible/blob/main/roles/edpm_nova/handlers/main.yml