It seems to me that there is a wrong behaviour when you configure a VIP with VMAC on RHOSP17.1.13. (VIP without VMAC are working fine)

I did these steps to reproduce the issue:

1. Create two instances on same network

2.  openstack server list --long
   +--------------------------------------+----------------+--------+------------+-------------+-------------------+------------+--------------------------------------+----------+-------------------+------------------------+------------+
   | ID                                   | Name           | Status | Task State | Power State | Networks          | Image Name | Image ID                             | Flavor   | Availability Zone | Host                   | Properties |
   +--------------------------------------+----------------+--------+------------+-------------+-------------------+------------+--------------------------------------+----------+-------------------+------------------------+------------+
   | a7bae725-59f7-455c-a52f-3bc58e0c6f02 | test-instance2 | ACTIVE | None       | Running     | net1=172.16.1.56  | rhel9      | 94d54cc8-7e62-4beb-be41-ca4c8f009448 | m1.small | nova              | compute-1.redhat.local |            |
   | 44b8a397-3927-491e-9eae-1318c979f677 | test-instance1 | ACTIVE | None       | Running     | net1=172.16.1.198 | rhel9      | 94d54cc8-7e62-4beb-be41-ca4c8f009448 | m1.small | nova              | compute-0.redhat.local |            |
   +--------------------------------------+----------------+--------+------------+-------------+-------------------+------------+--------------------------------------+----------+-------------------+------------------------+------------+
    

1. allow ICMP and SSH as default and add VIP and VMAC

2. openstack port set --allowed-address ip-address=172.16.1.218,mac-address=fa:16:3e:9f:0f:9a eae81688-94d9-4694-b43f-0cc95bee7eeb
   openstack port set --allowed-address ip-address=172.16.1.218,mac-address=fa:16:3e:9f:0f:9a 32c00f4f-abec-446d-a0a3-8fef7843ce12 

1. configure VIP and VMAC on guest

2. - For Node 1 (IP 172.16.1.198)
   # Bridge and Main IP Setup
   ip link add name br0 type bridge
   ip link set dev eth0 master br0
   ip addr flush dev eth0
   ip addr add 172.16.1.198/24 dev br0
   ip link set eth0 up
   ip link set br0 up
   ip route add default via 172.16.1.1 dev br0# 2. VIP Setup (Macvlan on br0)
   ip link add link br0 name eth0.vip address fa:16:3e:9f:0f:9a type macvlan mode bridge
   ip addr add 172.16.1.218/24 dev eth0.vip
   # ip link set eth0.vip up 
   # ARP Flux Fixes (Stop the Bridge from stealing VIP traffic)
   echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
   echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
   echo 1 > /proc/sys/net/ipv4/conf/br0/arp_ignore
   echo 2 > /proc/sys/net/ipv4/conf/br0/arp_announce
   echo 0 > /proc/sys/net/ipv4/conf/eth0.vip/arp_ignore
   echo 0 > /proc/sys/net/ipv4/conf/eth0.vip/rp_filter
   echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
   echo 0 > /proc/sys/net/ipv4/conf/br0/rp_filter
   
   - For Node 2 (IP 172.16.1.56):
   # Bridge and Main IP Setup
   ip link add name br0 type bridge
   ip link set dev eth0 master br0
   ip addr flush dev eth0
   ip addr add 172.16.1.56/24 dev br0
   ip link set eth0 up
   ip link set br0 up
   ip route add default via 172.16.1.1 dev br0# 2. VIP Setup (Macvlan on br0)
   ip link add link br0 name eth0.vip address fa:16:3e:9f:0f:9a type macvlan mode bridge
   ip addr add 172.16.1.218/24 dev eth0.vip
   ip link set eth0.vip up
   
   # ARP Flux Fixes (Stop the Bridge from stealing VIP traffic)
   echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
   echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
   echo 1 > /proc/sys/net/ipv4/conf/br0/arp_ignore
   echo 2 > /proc/sys/net/ipv4/conf/br0/arp_announce
   echo 0 > /proc/sys/net/ipv4/conf/eth0.vip/arp_ignore
   echo 0 > /proc/sys/net/ipv4/conf/eth0.vip/rp_filter
   echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
   echo 0 > /proc/sys/net/ipv4/conf/br0/rp_filter# 4. Routing Fixes
   # I'm not sure this rule are needed, but I added and start working but then I set down and up in the other and did not work
   ip rule add from 172.16.1.218 table 200
   ip route add 172.16.1.0/24 dev eth0.vip table 200
   ip route add default via 172.16.1.1 dev eth0.vip table 200 

1. Set up and down the interface 2-3 times, and you will see icmp reaching only one node and not the other, plus arp response from one of the two seems ignored

2. - For Node 2 (IP 172.16.1.56):
   ip link set eth0.vip down
   - For Node 1 (IP 172.16.1.198) 
   ip link set eth0.vip up 
   #do some ping test
   ip link set eth0.vip down
   - For Node 2 (IP 172.16.1.56):  
   ip link set eth0.vip up 
   #do some ping test
   ip link set eth0.vip down
   - For Node 1 (IP 172.16.1.198)
   ip link set eth0.vip up 
   #do some ping test
   ip link set eth0.vip down
   - For Node 2 (IP 172.16.1.56):  
   ip link set eth0.vip up 
   #do some ping test
   
    

    

From what I can see the reply reaches the tap interface but icmp were not routed to how own the VMAC.

[tripleo-admin@compute-0 ~]$ sudo tcpdump -nni tap32c00f4f-ab arp or icmp               tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tap32c00f4f-ab, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:31:40.913195 IP 172.16.1.204 > 172.16.1.218: ICMP echo request, id 64257, seq 0, length 64
13:31:40.925936 IP 172.16.1.218 > 172.16.1.204: ICMP echo reply, id 64257, seq 0, length 64
13:31:45.914006 ARP, Request who-has 172.16.1.218 tell 172.16.1.204, length 28
13:31:45.914483 ARP, Reply 172.16.1.218 is-at fa:16:3e:9f:0f:9a, length 28
13:31:45.983844 ARP, Request who-has 172.16.1.204 tell 172.16.1.198, length 28
13:31:45.984348 ARP, Reply 172.16.1.204 is-at fa:16:3e:ce:1d:05, length 28
 

I also tried to disable the security rule on ports, but it did not change.