Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-2464

Test Coverage: The guest run ping test failed after do netperf test with 8193byte on rhel9.2

      This task is tracking the test case writing activities to cover the bug described below.

      Description of problem:
      The guest run ping test failed after do netperf test with 8193byte on rhel9.2

      Version-Release number of selected component (if applicable):
      [root@dell-per750-37 ~]# uname -r
      5.14.0-284.10.1.el9_2.x86_64
      [root@dell-per750-37 ~]# rpm -qa|grep openvswitch
      openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
      openvswitch2.17-2.17.0-87.el9fdp.x86_64

      How reproducible:

      Steps to Reproduce:
      SERVER: dell750-37
      CLIENT: dell740-55
      ixgbe dell750-37 <connect> i40e dell740-55
      On server side:
      1. Build following ovs topo:
      bind the pf to vfi-pci and add the ovs bridge. and add dpdkvhostuserclient0 to ovs bridge too.
      [root@dell-per750-37 ~]# ovs-vsctl show
      27e5be3e-c9e3-4941-8321-aeda5f851dd7
      Bridge ovsbr0
      datapath_type: netdev
      Port dpdk0
      Interface dpdk0
      type: dpdk
      options:

      {dpdk-devargs="0000:17:00.0"}

      Port ovsbr0
      Interface ovsbr0
      type: internal
      Port dpdkvhostuserclient0
      Interface dpdkvhostuserclient0
      type: dpdkvhostuserclient
      options:

      {vhost-server-path="/tmp/dpdkvhostuserclient0"}

      ovs_version: "2.17.7"

      2. configure mtu to 9000
      ovs-vsctl set Interface dpdkvhostuserclient0 mtu_request=9000
      ovs-vsctl set Interface dpdk0 mtu_request=9000

      2. Start guest with following xml
      [root@dell-per750-37 ~]# virsh dumpxml master
      <domain type='kvm' id='4'>
      <name>master</name>
      <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
      <memory unit='KiB'>5242880</memory>
      <currentMemory unit='KiB'>5242880</currentMemory>
      <memoryBacking>
      <hugepages>
      <page size='1048576' unit='KiB'/>
      </hugepages>
      <access mode='shared'/>
      </memoryBacking>
      <vcpu placement='static'>3</vcpu>
      <cputune>
      <vcpupin vcpu='2' cpuset='0'/>
      <emulatorpin cpuset='2'/>
      </cputune>
      <numatune>
      <memory mode='strict' nodeset='0'/>
      </numatune>
      <resource>
      <partition>/machine</partition>
      </resource>
      <os>
      <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
      <boot dev='hd'/>
      </os>
      <features>
      <acpi/>
      <apic/>
      <pmu state='off'/>
      <vmport state='off'/>
      <ioapic driver='qemu'/>
      </features>
      <cpu mode='host-passthrough' check='none' migratable='on'>
      <feature policy='require' name='tsc-deadline'/>
      <numa>
      <cell id='0' cpus='0-2' memory='5242880' unit='KiB' memAccess='shared'/>
      </numa>
      </cpu>
      <clock offset='utc'>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='hpet' present='no'/>
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>restart</on_crash>
      <pm>
      <suspend-to-mem enabled='no'/>
      <suspend-to-disk enabled='no'/>
      </pm>
      <devices>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/master.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </disk>
      <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
      </controller>
      <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
      </controller>
      <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </controller>
      <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </controller>
      <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </controller>
      <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </controller>
      <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </controller>
      <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      </controller>
      <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
      </controller>
      <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:e8'/>
      <source type='unix' path='/tmp/dpdkvhostuserclient0' mode='server'/>
      <target dev='dpdkvhostuserclient0'/>
      <model type='virtio'/>
      <driver name='vhost' iommu='on' ats='on'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </interface>
      <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet3'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </interface>
      <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
      <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
      </serial>
      <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target type='isa-serial' port='1'>
      <model name='isa-serial'/>
      </target>
      <alias name='serial1'/>
      </serial>
      <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      </console>
      <input type='mouse' bus='ps2'>
      <alias name='input0'/>
      </input>
      <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
      </input>
      <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
      </graphics>
      <audio id='1' type='none'/>
      <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </video>
      <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </memballoon>
      <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
      <alias name='iommu0'/>
      </iommu>
      </devices>
      <seclabel type='dynamic' model='selinux' relabel='yes'>
      <label>system_u:system_r:svirt_t:s0:c421,c512</label>
      <imagelabel>system_u:object_r:svirt_image_t:s0:c421,c512</imagelabel>
      </seclabel>
      <seclabel type='dynamic' model='dac' relabel='yes'>
      <label>+107:+1001</label>
      <imagelabel>+107:+1001</imagelabel>
      </seclabel>
      </domain>

      3. Configure the port inside guest
      ip address: 20.0.0.1
      mtu: 9000
      [root@localhost ~]# ip a
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
      2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
      link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff
      inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
      valid_lft 2682sec preferred_lft 2682sec
      inet6 fe80::bc2:f054:b47a:3336/64 scope link noprefixroute
      valid_lft forever preferred_lft forever
      3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
      link/ether 52:54:00:11:8f:e8 brd ff:ff:ff:ff:ff:ff
      inet 20.0.0.1/24 scope global enp3s0
      valid_lft forever preferred_lft forever
      inet6 fe80::43f0:cab:404f:8586/64 scope link noprefixroute
      valid_lft forever preferred_lft forever

      On client side:
      configure ip address and mtu 9000 on client:
      [root@dell-per740-55 ~]# ip a
      6: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
      link/ether f8:f2:1e:bb:cf:80 brd ff:ff:ff:ff:ff:ff
      altname enp94s0f0
      inet 20.0.0.3/24 scope global ens3f0
      valid_lft forever preferred_lft forever

      Inside server's guest ping client's port address:
      [root@localhost ~]# ping 20.0.0.3
      PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
      64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=0.060 ms
      64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.135 ms
      64 bytes from 20.0.0.3: icmp_seq=4 ttl=64 time=0.062 ms

      Start netserver on client side:
      [root@dell-per740-55 ~]# ps aux|grep netserver
      root 2144 0.0 0.0 3840 1352 ? Ss 03:25 0:00 netserver
      root 2291 0.0 0.0 6408 2300 pts/1 S+ 05:05 0:00 grep --color=auto netserver

      Do netperf test inside guest of server,
      [root@localhost ~]# netperf -t UDP_STREAM -H 20.0.0.3 – -m 1025
      MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 20.0.0.3 () port 0 AF_INET
      Socket Message Elapsed Messages
      Size Size Time Okay Errors Throughput
      bytes bytes secs # # 10^6bits/sec

      212992 1025 10.00 5205532 0 4268.52
      212992 10.00 5110575 4190.66

      [root@localhost ~]# netperf -t UDP_STREAM -H 20.0.0.3 – -m 8193
      MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 20.0.0.3 () port 0 AF_INET
      Socket Message Elapsed Messages
      Size Size Time Okay Errors Throughput
      bytes bytes secs # # 10^6bits/sec

      212992 8193 14.20 476933 0 2200.71
      212992 14.20 0 0.00

      Run ping again:
      [root@localhost ~]# ping 20.0.0.3
      PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
      ping: sendmsg: No buffer space available
      ping: sendmsg: No buffer space available

      Actual results:
      netperf test with 8193byte failed.
      And after do the netperf test with 8193byte, the guest cannot ping the client ip address anymore.

      Expected results:
      After do the netperf test with 8193byte, the guest should ping the client ip address successfully.
      When restart the guest, the guest still ping the client ip failed.
      [root@localhost ~]# ping 20.0.0.3
      PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
      From 20.0.0.1 icmp_seq=1 Destination Host Unreachable

      When restart the openvswitch service, the guest ping the client ip successfully.

      Additional info:
      https://beaker.engineering.redhat.com/jobs/7892906
      For another the i40e<->i40e test env, it also has this issue.

              ovsdpdk-triage ovsdpdk triage
              nstbot NST Bot
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: