QE found a bug related to multicast on ovn-kubernetes https://issues.redhat.com/browse/OCPBUGS-34778 . The change happened between 4.16.0-ec.6 (April 29th 2024, works) and 4.16.0-rc.0 (May 6th  2024, fails). We see an OVN bump from 23.09.0-139 to between these dates: https://github.com/openshift/ovn-kubernetes/commit/72e5b28a12e4df3d08d05d30662327365cc39322

https://issues.redhat.com/browse/OCPBUGS-34778?focusedId=24907874&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24907874

Basically when a pod tries to send a multicast packet to a pod on a different node, the packet gets dropped somewhere before it can get to the geneve tunnel.

An ovs flow blocks this traffic:

recirc_id(0),in_port(6),skb_mark(0),ct_state(-new-est-rel-rpl-inv-trk-dnat),ct_mark(0/0xf),eth(src=0a:58:0a:f4:02:03,dst=01:00:5e:40:61:69),eth_type(0x0800),ipv4(src=10.244.2.3,dst=239.192.97.105,proto=17,ttl=64,frag=no), packets:1528, bytes:169608, used:0.576s, actions:drop

Please find the relevant trace and dbs in the attached archive.