Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-73891

No RARP packets on the destination after migration [rhel-9.6]

XMLWordPrintable

    • qemu-kvm-9.1.0-15.el9
    • Yes
    • Important
    • CustomerScenariosInitiative
    • 2
    • rhel-sst-virtualization-networking
    • ssg_virtualization
    • 26
    • 28
    • 1
    • QE ack
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • Virtio-Net Sprint 9 - January, Virtio-Net Sprint 10 - Feb
    • Unspecified
    • Unspecified
    • None

      What were you trying to do that didn't work?
      No RARP packet on the destination after migration

      Please provide the package NVR for which bug is seen:
      kernel-5.14.0-545.el9.x86_64 && qemu-kvm-9.1.0-7.el9.x86_64

      How reproducible:
      100%

      Steps to reproduce
      1. Boot a guest with a tap device on src host:

      -device '{"driver":"virtio-net-pci","mac":"52:54:0b:4c:d5:db","id":"net0","netdev":"tap0","bus":"root2","addr":"0x0"}' \
-netdev tap,id=tap0,vhost=on,ifname=tap3,script=/bin/true \

      2. Turn on tap3, and monitor tap3 via tcpdump on src host

      # ip link set dev tap3 up
# tcpdump -n -i tap3 not ip6 > /dev/null 2>&1 &

      3. Boot a listening qemu on dst host with tap4

      -device '{"driver":"virtio-net-pci","mac":"52:54:0b:4c:d5:db","id":"net0","netdev":"tap0","bus":"root2","addr":"0x0"}' \
-netdev tap,id=tap0,vhost=on,ifname=tap4,script=/bin/true \

      4. In guest

      # systemctl stop NetworkManager
# ip addr add dev eth0 192.168.10.10

      5.Turn on tap4 and monitor tap4 via tcpdump on dst host

      # ip link set dev tap4 up
# tcpdump -n -i tap4 not ip6 -ttttt > /home/ipa/RHEL_169571_tcpdump.log

      6. Start migration from the src to dst host
      7. After migration, Check the ARP/RARP packets emitted from step 5 on dst host

      Expected results
      ARP/RARP emitted successfully on dst host after migration

      Actual results

      No RARP packets emitted on dst host after migration
2025-01-08-06:06:05:  00:00:00.000000 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-08-06:06:05:  00:00:00.000103 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-08-06:06:05:  00:00:00.049991 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-08-06:06:05:  00:00:00.050063 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-08-06:06:05:  00:00:00.248043 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-08-06:06:05:  00:00:00.248050 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-08-06:06:05:  00:00:00.497913 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-08-06:06:05:  00:00:00.497919 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-08-06:06:05:  00:00:00.847915 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-08-06:06:05:  00:00:00.847924 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-08-06:06:05:  00:00:30.007337 IP 192.168.10.10.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 (Cache flush) AAAA fe80::5054:bff:fe4c:d5db, (Cache flush) PTR linux.local., (Cache flush) A 192.168.10.10, (Cache flush) PTR linux.local. (189)
2025-01-08-06:06:05:  00:00:30.009717 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-08-06:06:05:  00:00:30.290733 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)

      Additional info:
      The full qemu cmd from the src:

      /usr/libexec/qemu-kvm  \
-name "mouse-vm" \
-sandbox on \
-machine q35,memory-backend=pc.ram,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
-cpu SapphireRapids,dtes64=on,vmx=on,pdcm=on,hypervisor=on,ds=on,ss=on,tsc-adjust=on,waitpkg=on,cldemote=on,movdiri=on,movdir64b=on,md-clear=on,stibp=on,flush-l1d=on,invtsc=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,tsx-ctrl=on,sbdr-ssdp-no=on,fbsdp-no=on,psdp-no=on,gds-no=on,rfds-no=on,vmx-ins-outs=on,vmx-true-ctls=on,vmx-store-lma=on,vmx-activity-hlt=on,vmx-activity-wait-sipi=on,vmx-vmwrite-vmexit-fields=on,vmx-apicv-xapic=on,vmx-ept=on,vmx-desc-exit=on,vmx-rdtscp-exit=on,vmx-apicv-x2apic=on,vmx-vpid=on,vmx-wbinvd-exit=on,vmx-unrestricted-guest=on,vmx-apicv-register=on,vmx-apicv-vid=on,vmx-rdrand-exit=on,vmx-invpcid-exit=on,vmx-vmfunc=on,vmx-shadow-vmcs=on,vmx-rdseed-exit=on,vmx-pml=on,vmx-xsaves=on,vmx-tsc-scaling=on,vmx-enable-user-wait-pause=on,vmx-ept-execonly=on,vmx-page-walk-4=on,vmx-page-walk-5=on,vmx-ept-2mb=on,vmx-ept-1gb=on,vmx-invept=on,vmx-eptad=on,vmx-invept-single-context=on,vmx-invept-all-context=on,vmx-invvpid=on,vmx-invvpid-single-addr=on,vmx-invvpid-all-context=on,vmx-invept-single-context-noglobals=on,vmx-intr-exit=on,vmx-nmi-exit=on,vmx-vnmi=on,vmx-preemption-timer=on,vmx-posted-intr=on,vmx-vintr-pending=on,vmx-tsc-offset=on,vmx-hlt-exit=on,vmx-invlpg-exit=on,vmx-mwait-exit=on,vmx-rdpmc-exit=on,vmx-rdtsc-exit=on,vmx-cr3-load-noexit=on,vmx-cr3-store-noexit=on,vmx-cr8-load-exit=on,vmx-cr8-store-exit=on,vmx-flexpriority=on,vmx-vnmi-pending=on,vmx-movdr-exit=on,vmx-io-exit=on,vmx-io-bitmap=on,vmx-mtf=on,vmx-msr-bitmap=on,vmx-monitor-exit=on,vmx-pause-exit=on,vmx-secondary-ctls=on,vmx-exit-nosave-debugctl=on,vmx-exit-load-perf-global-ctrl=on,vmx-exit-ack-intr=on,vmx-exit-save-pat=on,vmx-exit-load-pat=on,vmx-exit-save-efer=on,vmx-exit-load-efer=on,vmx-exit-save-preemption-timer=on,vmx-entry-noload-debugctl=on,vmx-entry-ia32e-mode=on,vmx-entry-load-perf-global-ctrl=on,vmx-entry-load-pat=on,vmx-entry-load-efer=on,vmx-eptp-switching=on,hle=off,rtm=off,taa-no=off,tsc-frequency=2000000000,host-phys-bits=on,host-phys-bits-limit=46 \
-nodefaults  \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1,server=on,wait=off \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor,server=on,wait=off \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device '{"driver":"pcie-root-port","id":"root0","multifunction":true,"bus":"pcie.0","addr":"0x2","chassis":1}' \
-device '{"driver":"pcie-root-port","id":"root1","port":11,"addr":"0x2.0x1","bus":"pcie.0","chassis":2}' \
-device '{"driver":"pcie-root-port","id":"root2","port":12,"addr":"0x2.0x2","bus":"pcie.0","chassis":3}' \
-device '{"driver":"pcie-root-port","id":"root3","port":13,"addr":"0x2.0x3","bus":"pcie.0","chassis":4}' \
-device '{"driver":"pcie-root-port","id":"root4","port":14,"addr":"0x2.0x4","bus":"pcie.0","chassis":5}' \
-device '{"driver":"pcie-root-port","id":"root5","port":15,"addr":"0x2.0x5","bus":"pcie.0","chassis":6}' \
-device '{"driver":"pcie-root-port","id":"root6","port":16,"addr":"0x2.0x6","bus":"pcie.0","chassis":7}' \
-device '{"driver":"pcie-root-port","id":"root7","port":17,"addr":"0x2.0x7","bus":"pcie.0","chassis":8}' \
-device '{"driver":"pcie-root-port","id":"extra_root0","multifunction":true,"bus":"pcie.0","addr":"0x3","chassis":21}' \
-device '{"driver":"pcie-root-port","id":"extra_root1","port":21,"addr":"0x3.0x1","bus":"pcie.0","chassis":22}' \
-device '{"driver":"pcie-root-port","id":"extra_root2","port":22,"addr":"0x3.0x2","bus":"pcie.0","chassis":23}' \
-device '{"driver":"nec-usb-xhci","id":"usb1","bus":"root0","addr":"0x0"}' \
-device '{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci0","bus":"root1","addr":"0x0"}' \
-device '{"driver":"scsi-hd","id":"image1","device_id":"drive-image1","drive":"drive_image1","bus":"virtio_scsi_pci0.0","channel":0,"scsi-id":0,"lun":0,"bootindex":0,"write-cache":"on"}' \
-device '{"driver":"virtio-net-pci","mac":"52:54:0b:4c:d5:db","id":"net0","netdev":"tap0","bus":"root2","addr":"0x0"}' \
-device '{"driver":"usb-tablet","id":"usb-tablet1","bus":"usb1.0","port":"1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"root3","addr":"0x0"}' \
-device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-blockdev '{"driver":"file","auto-read-only":true,"discard":"unmap","aio":"threads","cache":{"direct":true,"no-flush":false},"filename":"/mnt/xiaohli/rhel960-64-virtio-scsi-ovmf.qcow2","node-name":"drive_sys1"}' \
-blockdev '{"driver":"qcow2","node-name":"drive_image1","read-only":false,"cache":{"direct":true,"no-flush":false},"file":"drive_sys1"}' \
-blockdev '{"node-name":"file_ovmf_code","driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"drive_ovmf_code","driver":"raw","read-only":true,"file":"file_ovmf_code"}' \
-blockdev '{"node-name":"file_ovmf_vars","driver":"file","filename":"/mnt/xiaohli/rhel960-64-virtio-scsi-ovmf.qcow2_VARS.fd","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"drive_ovmf_vars","driver":"raw","read-only":false,"file":"file_ovmf_vars"}' \
-netdev tap,id=tap0,vhost=on,ifname=tap3,script=/bin/true \
-m 4096 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296}' \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-vnc :10 \
-rtc base=utc,clock=host \
-boot menu=off,strict=off,order=cdn,once=c \
-enable-kvm  \
-qmp tcp:0:3333,server=on,wait=off \
-qmp tcp:0:9999,server=on,wait=off \
-qmp tcp:0:9888,server=on,wait=off \
-serial tcp:0:4444,server=on,wait=off \
-monitor stdio \
-msg timestamp=on \

            [RHEL-73891] No RARP packets on the destination after migration [rhel-9.6]

            Xiaohui Li added a comment -

            Test pass on RHEL 9.6.0 (kernel-5.14.0-569.el9.x86_64 && qemu-kvm-9.1.0-15.el9.x86_64), after migration, get ARP, RARP packets.

            2025-02-20-02:04:04:  00:00:00.000000 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.003382 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.003737 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.039726 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.053729 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.053793 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.189842 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.269269 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.269302 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.439897 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.518817 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.518844 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.789962 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.868811 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-02-20-02:04:04:  Matched.
2025-02-20-02:04:04:  00:00:00.868840 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)

            Xiaohui Li added a comment - Test pass on RHEL 9.6.0 (kernel-5.14.0-569.el9.x86_64 && qemu-kvm-9.1.0-15.el9.x86_64), after migration, get ARP, RARP packets. 2025-02-20-02:04:04: 00:00:00.000000 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.003382 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.003737 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.039726 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.053729 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.053793 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.189842 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.269269 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.269302 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.439897 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.518817 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.518844 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.789962 ARP, Reverse Request who-is 52:54:d3:58:b1:d3 tell 52:54:d3:58:b1:d3, length 46 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.868811 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-02-20-02:04:04: Matched. 2025-02-20-02:04:04: 00:00:00.868840 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)

            Jon Maloy added a comment -

            Fix included in qemu-kvm-9.1.0-15.el9

            Fixed by merge request 'net: Fix announce_self' ( https://gitlab.com/redhat/centos-stream/src/qemu-kvm/-/merge_requests/338 )

            Jon Maloy added a comment - Fix included in qemu-kvm-9.1.0-15.el9 Fixed by merge request 'net: Fix announce_self' ( https://gitlab.com/redhat/centos-stream/src/qemu-kvm/-/merge_requests/338 )

            Xiaohui Li added a comment -

            lvivier@redhat.com , done, thanks for the heads up.

            Xiaohui Li added a comment - lvivier@redhat.com , done, thanks for the heads up.

            Laurent Vivier added a comment -

            rhn-support-xiaohli 

            could you set Target End (ITM)?

            Thanks

            Laurent Vivier added a comment - rhn-support-xiaohli   could you set Target End (ITM)? Thanks

            Xiaohui Li added a comment -

            This issue should be a regression from qemu-kvm-9.1.0-1.el9.x86_64

            Xiaohui Li added a comment - This issue should be a regression from qemu-kvm-9.1.0-1.el9.x86_64

            Xiaohui Li added a comment - - edited

            hi lvivier@redhat.com, thanks for the quick fix.

            Could you try following build?

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=66401515

            I ran two tests on this scratch build, test results as below:

            1) if I didn't add "guest_announce": false, RHEL-169571 case test passed, ARP/RARP emitted:

            2025-01-17-13:41:28: [root@host ~]# cat /home/ipa/RHEL_169571_tcpdump.log
2025-01-17-13:41:28: dropped privs to tcpdump
2025-01-17-13:41:28: tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
2025-01-17-13:41:28: listening on tap4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
2025-01-17-13:41:28:  00:00:00.000000 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46
2025-01-17-13:41:28:  00:00:00.002021 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-17-13:41:28:  00:00:00.002231 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-17-13:41:28:  00:00:00.049569 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46
2025-01-17-13:41:28:  00:00:00.051390 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-17-13:41:28:  00:00:00.051437 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-17-13:41:28:  00:00:00.212041 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46
2025-01-17-13:41:28:  00:00:00.212176 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-17-13:41:28:  00:00:00.212202 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-17-13:41:28:  00:00:00.461549 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46
2025-01-17-13:41:28:  00:00:00.461633 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-17-13:41:28:  00:00:00.461658 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-17-13:41:28:  00:00:00.811634 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46
2025-01-17-13:41:28:  00:00:00.811699 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28
2025-01-17-13:41:28:  00:00:00.811716 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-17-13:41:28:  00:00:30.012763 IP 192.168.10.10.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 (Cache flush) AAAA fe80::5054:31ff:fe39:7ba4, (Cache flush) PTR linux.local., (Cache flush) A 192.168.10.10, (Cache flush) PTR linux.local. (189)
2025-01-17-13:41:28:  00:00:30.015049 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-17-13:41:28:  00:00:30.929035 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)

            2) added "guest_announce": false, 

            2025-01-17-13:35:30: [root@host ~]# cat /home/ipa/RHEL_169571_tcpdump.log
2025-01-17-13:35:30: dropped privs to tcpdump
2025-01-17-13:35:30: tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
2025-01-17-13:35:30: listening on tap4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
2025-01-17-13:35:30:  00:00:00.000000 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46
2025-01-17-13:35:30:  00:00:00.049697 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46
2025-01-17-13:35:30:  00:00:00.250869 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46
2025-01-17-13:35:30:  00:00:00.500893 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46
2025-01-17-13:35:30:  00:00:00.850987 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46
2025-01-17-13:35:30:  00:00:30.007802 IP 192.168.10.10.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 (Cache flush) AAAA fe80::5054:eaff:fe3c:9b7, (Cache flush) PTR linux.local., (Cache flush) A 192.168.10.10, (Cache flush) PTR linux.local. (189)
2025-01-17-13:35:30:  00:00:30.009915 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)
2025-01-17-13:35:30:  00:00:30.856924 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)

            Xiaohui Li added a comment - - edited hi lvivier@redhat.com , thanks for the quick fix. Could you try following build? https: //brewweb.engineering.redhat.com/brew/taskinfo?taskID=66401515 I ran two tests on this scratch build, test results as below: 1) if I didn't add "guest_announce": false, RHEL-169571 case test passed, ARP/RARP emitted: 2025-01-17-13:41:28: [root@host ~]# cat /home/ipa/RHEL_169571_tcpdump.log 2025-01-17-13:41:28: dropped privs to tcpdump 2025-01-17-13:41:28: tcpdump: verbose output suppressed, use -v[v]... for full protocol decode 2025-01-17-13:41:28: listening on tap4, link-type EN10MB (Ethernet), snapshot length 262144 bytes 2025-01-17-13:41:28: 00:00:00.000000 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46 2025-01-17-13:41:28: 00:00:00.002021 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-01-17-13:41:28: 00:00:00.002231 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-01-17-13:41:28: 00:00:00.049569 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46 2025-01-17-13:41:28: 00:00:00.051390 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-01-17-13:41:28: 00:00:00.051437 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-01-17-13:41:28: 00:00:00.212041 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46 2025-01-17-13:41:28: 00:00:00.212176 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-01-17-13:41:28: 00:00:00.212202 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-01-17-13:41:28: 00:00:00.461549 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46 2025-01-17-13:41:28: 00:00:00.461633 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-01-17-13:41:28: 00:00:00.461658 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-01-17-13:41:28: 00:00:00.811634 ARP, Reverse Request who-is 52:54:31:39:7b:a4 tell 52:54:31:39:7b:a4, length 46 2025-01-17-13:41:28: 00:00:00.811699 ARP, Request who-has 192.168.10.10 tell 192.168.10.10, length 28 2025-01-17-13:41:28: 00:00:00.811716 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-01-17-13:41:28: 00:00:30.012763 IP 192.168.10.10.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 (Cache flush) AAAA fe80::5054:31ff:fe39:7ba4, (Cache flush) PTR linux.local., (Cache flush) A 192.168.10.10, (Cache flush) PTR linux.local. (189) 2025-01-17-13:41:28: 00:00:30.015049 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-01-17-13:41:28: 00:00:30.929035 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2) added "guest_announce": false, 2025-01-17-13:35:30: [root@host ~]# cat /home/ipa/RHEL_169571_tcpdump.log 2025-01-17-13:35:30: dropped privs to tcpdump 2025-01-17-13:35:30: tcpdump: verbose output suppressed, use -v[v]... for full protocol decode 2025-01-17-13:35:30: listening on tap4, link-type EN10MB (Ethernet), snapshot length 262144 bytes 2025-01-17-13:35:30: 00:00:00.000000 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46 2025-01-17-13:35:30: 00:00:00.049697 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46 2025-01-17-13:35:30: 00:00:00.250869 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46 2025-01-17-13:35:30: 00:00:00.500893 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46 2025-01-17-13:35:30: 00:00:00.850987 ARP, Reverse Request who-is 52:54:ea:3c:09:b7 tell 52:54:ea:3c:09:b7, length 46 2025-01-17-13:35:30: 00:00:30.007802 IP 192.168.10.10.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 (Cache flush) AAAA fe80::5054:eaff:fe3c:9b7, (Cache flush) PTR linux.local., (Cache flush) A 192.168.10.10, (Cache flush) PTR linux.local. (189) 2025-01-17-13:35:30: 00:00:30.009915 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s) 2025-01-17-13:35:30: 00:00:30.856924 IP 192.168.10.10 > 224.0.0.22: igmp v3 report, 1 group record(s)

            Laurent Vivier added a comment - - edited

            And this change fixes the problem with filter-dump object:

            diff --git a/net/dump.c b/net/dump.c
index d7dd2ce461de..140215aa1054 100644
--- a/net/dump.c
+++ b/net/dump.c
@@ -155,7 +155,8 @@ static ssize_t filter_dump_receive_iov(NetFilterState *nf, NetClientState *sndr,
 {
     NetFilterDumpState *nfds = FILTER_DUMP(nf);
 
-    dump_receive_iov(&nfds->ds, iov, iovcnt, qemu_get_vnet_hdr_len(nf->netdev));
+    dump_receive_iov(&nfds->ds, iov, iovcnt, flags & QEMU_NET_PACKET_FLAG_RAW ?
+                     0 : qemu_get_vnet_hdr_len(nf->netdev));
     return 0;
 }

            Laurent Vivier added a comment - - edited And this change fixes the problem with filter-dump object: diff --git a/net/dump.c b/net/dump.c index d7dd2ce461de..140215aa1054 100644 --- a/net/dump.c +++ b/net/dump.c @@ -155,7 +155,8 @@ static ssize_t filter_dump_receive_iov(NetFilterState *nf, NetClientState *sndr, { NetFilterDumpState *nfds = FILTER_DUMP(nf); - dump_receive_iov(&nfds->ds, iov, iovcnt, qemu_get_vnet_hdr_len(nf->netdev)); + dump_receive_iov(&nfds->ds, iov, iovcnt, flags & QEMU_NET_PACKET_FLAG_RAW ? + 0 : qemu_get_vnet_hdr_len(nf->netdev)); return 0; }

            Laurent Vivier added a comment -

            There is a bug QEMU in the only case of the announce_self (and another one with filter-dump, I think).
            The following change fixes the problem for me

            diff --git a/net/net.c b/net/net.c
index c1bb19a52373..9cded70dde74 100644
--- a/net/net.c
+++ b/net/net.c
@@ -822,6 +822,7 @@ static ssize_t qemu_deliver_packet_iov(NetClientState *sender,
         iov_copy[0].iov_len =  nc->vnet_hdr_len;
         memcpy(&iov_copy[1], iov, iovcnt * sizeof(*iov));
         iov = iov_copy;
+        iovcnt++;
     }
 
     if (nc->info->receive_iov) {

            Laurent Vivier added a comment - There is a bug QEMU in the only case of the announce_self (and another one with filter-dump, I think). The following change fixes the problem for me diff --git a/net/net.c b/net/net.c index c1bb19a52373..9cded70dde74 100644 --- a/net/net.c +++ b/net/net.c @@ -822,6 +822,7 @@ static ssize_t qemu_deliver_packet_iov(NetClientState *sender, iov_copy[0].iov_len = nc->vnet_hdr_len; memcpy(&iov_copy[1], iov, iovcnt * sizeof (*iov)); iov = iov_copy; + iovcnt++; } if (nc->info->receive_iov) {

            Laurent Vivier added a comment -

            The RARP packet is shifted by the size of the virtio_net_hdr_mrg_rxbuf structure, 12 bytes.

            Laurent Vivier added a comment - The RARP packet is shifted by the size of the virtio_net_hdr_mrg_rxbuf structure, 12 bytes.

            Laurent Vivier added a comment -

            Thank you rhn-support-xiaohli 

            This is weird because in this case we should have RARP sent by QEMU (moreover if I read correctly the code we should always have RARP from QEMU and ARP from linux guest)

            I have collected the packet directly from the netdev interface and found the packet created by QEMU are invalid and I think discarded by the kernel networking stack.

            We can capture the packets directly from QEMU netdev interface using:

            ... -netdev tap,id=tap0,vhost=on,ifname=tap3 \
    -object filter-dump,netdev=tap0,file=log.pcap,id=pcap0

            and then:

            tcpdump -n  not ip6 -ttttt -r log.pcap

            reading from file log.pcap, link-type EN10MB (Ethernet), snapshot length 65536
dropped privs to tcpdump
00:00:00.000000 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 
0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./..
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 ....
00:00:00.049545 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 
0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./..
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 ....
00:00:00.199496 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 
0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./..
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 ....
00:00:00.449493 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 
0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./..
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 ....
00:00:00.799496 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 
0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./..
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 ....

            We can ask QEMU to send these packets without having to migrate using the monitor command "announce_self" (there is the same for QMP). It 's what I did.

            Laurent Vivier added a comment - Thank you rhn-support-xiaohli   This is weird because in this case we should have RARP sent by QEMU (moreover if I read correctly the code we should always have RARP from QEMU and ARP from linux guest) I have collected the packet directly from the netdev interface and found the packet created by QEMU are invalid and I think discarded by the kernel networking stack. We can capture the packets directly from QEMU netdev interface using: ... -netdev tap,id=tap0,vhost=on,ifname=tap3 \ -object filter-dump,netdev=tap0,file=log.pcap,id=pcap0 and then: tcpdump -n not ip6 -ttttt -r log.pcap reading from file log.pcap, link-type EN10MB (Ethernet), snapshot length 65536 dropped privs to tcpdump 00:00:00.000000 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./.. 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 .... 00:00:00.049545 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./.. 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 .... 00:00:00.199496 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./.. 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 .... 00:00:00.449493 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./.. 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 .... 00:00:00.799496 08:00:06:04:00:03 > 2e:2f:80:35:00:01, ethertype Unknown (0x9a2b), length 50: 0x0000: 2c2d 2e2f 0000 0000 9a2b 2c2d 2e2f 0000 ,{-}./.....+,{-}./.. 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 .... We can ask QEMU to send these packets without having to migrate using the monitor command "announce_self" (there is the same for QMP). It 's what I did.

              lvivier@redhat.com Laurent Vivier
              rhn-support-xiaohli Xiaohui Li
              virt-maint virt-maint
              Xiaohui Li Xiaohui Li
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated: