What were you trying to do that didn't work?

Multiple containers running nginx each with their own network, a single Traefik (reverse proxy) container connected to all networks and publishing 80,443. This does not work consistently unless rp_filter is explicitly set to 2 in sysctl. This is due to traffic usually leaving the proxy on a different interface than they enter on.

What impact?

As an end user this was quite difficult to troubleshoot as this is a typical design pattern and something that is expected to work. Eventually I did some nft tracing which led me to the asymetric routing issue and then to rp_filter. If everything is as intended, I would at the very least suggest updating the podman-network man page and/or rhel documentation to include a mention about rp_filter with regards to multihomed containers so that the solution could be more readilly discovered.

Please provide the package NVR for which the bug is seen: rhel 9.5, podman 5.2.2

How reproducible is this bug?: Always

Reproduction / Explanation:

rp_filter is set using the same priority in two different files, 50-redhat takes precedence here and interfaces are explicitly set to 1. This seems like a somewhat problematic / confusing design. See: https://access.redhat.com/solutions/6992481

grep rp_filter /usr/lib/sysctl.d/50-redhat.conf
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.*.rp_filter = 1
-net.ipv4.conf.all.rp_filter
 
grep rp_filter /usr/lib/sysctl.d/50-default.conf
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.*.rp_filter = 2
-net.ipv4.conf.all.rp_filter
However, podman itself tries to set the interfaces it creates to 2

$ podman --log-level debug run --network bridge quay.io/libpod/testimage:20241011 true |& grep rp_filter
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv4/conf/podman0/rp_filter to 2
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv4/conf/eth0/rp_filter to 2

This either doesn't work, or is changed immediately afterwards and the resulting interfaces have rp_filter set to 1.

Setting it explicity /etc/sysctl.d/99-rp_filter.conf

net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2

Does work, and podman created interfaces then have a value of 2.
 

See also: https://github.com/containers/podman/issues/26044