[3758692421] Upstream Reporter: dmurnane
Upstream issue status: Open
Upstream description:

Host is rocky linux 10.1, running kubernetes with cilium networking.

podman-5.6.0-8.el10_1.x86_64
netavark-1.16.0-1.el10.x86_64

On every other run, calling podman run fails with a veth creation error:

[dmurnane@dm5 ~]$ sudo podman run --rm -ti rockylinux/rockylinux:10
Error: netavark: create veth pair: Netlink error: Invalid argument (os error 22)
[dmurnane@dm5 ~]$ sudo podman run --rm -ti rockylinux/rockylinux:10
bash-5.2#
exit

This appears to work because the podman0 bridge is created with the default 1500 MTU but not cleaned up by the failing invocation, at which point netavark skips to creating the veth interface without trying to set an invalid MTU.

Running with --log-level=trace, the route dump is returning a host-scoped default route on the loopback interface as the first route

[TRACE netavark::network::netlink] send netlink packet: NetlinkMessage { header: NetlinkHeader { length: 28, message_type: 26, flags: 773, sequence_number: 2, port_number: 0 }, payload: InnerMessage(GetRoute(RouteMessage { header: RouteHeader { address_family: Unspec, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: Unspec, scope: Universe, kind: Unicast, flags: RouteFlags(0x0) }, attributes: [] })) }
[TRACE netavark::network::netlink] read netlink packet: NetlinkMessage { header: NetlinkHeader { length: 44, message_type: 24, flags: 2, sequence_number: 2, port_number: 7222 }, payload: InnerMessage(NewRoute(RouteMessage { header: RouteHeader { address_family: Inet, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 252, protocol: Kernel, scope: Host, kind: Local, flags: RouteFlags(0x0) }, attributes: [Table(2004), Oif(1)] })) }
[TRACE netavark::network::netlink] read netlink packet: NetlinkMessage { header: NetlinkHeader { length: 60, message_type: 24, flags: 2, sequence_number: 2, port_number: 7222 }, payload: InnerMessage(NewRoute(RouteMessage { header: RouteHeader { address_family: Inet, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: Static, scope: Universe, kind: Unicast, flags: RouteFlags(0x0) }, attributes: [Table(254), Priority(1), Gateway(Inet(10.0.44.1)), Oif(3)] })) }

This causes it to detect the loopback interface as the default route, and attempt to set the mtu to 65536, which fails with the invalid argument message.

[TRACE netavark::network::netlink] read netlink packet: NetlinkMessage { header: NetlinkHeader { length: 1440, message_type: 16, flags: 0, sequence_number: 3, port_number: 21222 }, payload: InnerMessage(NewLink(LinkMessage { header: LinkHeader { interface_family: Unspec, index: 1, link_layer_type: Loopback, flags: LinkFlags(Up | Loopback | Running | LowerUp), change_mask: LinkFlags(0x0) }, attributes: [IfName("lo"), <snip>
[DEBUG netavark::network::bridge] Using mtu 65536 from default route interface for the network
<snip>
[TRACE netavark::network::netlink] send netlink packet: NetlinkMessage { header: NetlinkHeader { length: 132, message_type: 16, flags: 1541, sequence_number: 8, port_number: 0 }, payload: InnerMessage(NewLink(LinkMessage { header: LinkHeader { interface_family: Unspec, index: 0, link_layer_type: Netrom, flags: LinkFlags(0x0), change_mask: LinkFlags(0x0) }, attributes: [LinkInfo([Kind(Veth), Data(Veth(Peer(LinkMessage { header: LinkHeader { interface_family: Unspec, index: 0, link_layer_type: Netrom, flags: LinkFlags(0x0), change_mask: LinkFlags(0x0) }, attributes: [LinkInfo([Kind(Veth)]), IfName("eth0"), Mtu(65536), NetNsFd(3)] })))]), Mtu(65536), Controller(39)] })) }
[TRACE netavark::network::netlink] read netlink packet: NetlinkMessage { header: NetlinkHeader { length: 152, message_type: 2, flags: 0, sequence_number: 8, port_number: 21222 }, payload: Error(ErrorMessage { code: Some(-22), header: [132, 0, 0, 0, 16, 0, 5, 6, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 18, 0, 9, 0, 1, 0, 118, 101, 116, 104, 0, 0, 0, 0, 68, 0, 2, 0, 64, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16, 0, 18, 0, 9, 0, 1, 0, 118, 101, 116, 104, 0, 0, 0, 0, 9, 0, 3, 0, 101, 116, 104, 48, 0, 0, 0, 0, 8, 0, 4, 0, 0, 0, 1, 0, 8, 0, 28, 0, 3, 0, 0, 0, 8, 0, 4, 0, 0, 0, 1, 0, 8, 0, 10, 0, 39, 0, 0, 0] }) }

Route table:

[dmurnane@dm5 ~]$ ip r list table 2004
local default dev lo proto kernel scope host
[dmurnane@dm5 ~]$ ip r list table 254
default via 10.0.44.1 dev ens19 proto static metric 1
10.0.44.0/24 dev ens19 proto kernel scope link src 10.0.44.101 metric 101
10.1.0.0/24 dev ens18 proto kernel scope link src 10.1.0.5 metric 100
10.101.0.0/24 via 10.101.0.162 dev cilium_host proto kernel src 10.101.0.162
10.101.0.0/16 dev ens18 proto static scope link metric 1
10.101.0.162 dev cilium_host proto kernel scope link
10.101.1.0/24 via 10.101.0.162 dev cilium_host proto kernel src 10.101.0.162 mtu 8900
10.201.0.0/16 dev ens18 proto static scope link metric 1

If podman is run using a network with an MTU configured (either adding another network or patching the default podman bridge network), the problem does not occur.

If podman is run as a user, the problem does not occur.