Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-823

The guest run ping test failed after do netperf test with 8193byte on rhel9.2

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • openvswitch2.17
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • rhel-sst-network-fastdatapath
    • ssg_networking

      Description of problem:
      The guest run ping test failed after do netperf test with 8193byte on rhel9.2

      Version-Release number of selected component (if applicable):
      [root@dell-per750-37 ~]# uname -r
      5.14.0-284.10.1.el9_2.x86_64
      [root@dell-per750-37 ~]# rpm -qa|grep openvswitch
      openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
      openvswitch2.17-2.17.0-87.el9fdp.x86_64

      How reproducible:

      Steps to Reproduce:
      SERVER: dell750-37
      CLIENT: dell740-55
      ixgbe dell750-37 <connect> i40e dell740-55
      On server side:
      1. Build following ovs topo:
      bind the pf to vfi-pci and add the ovs bridge. and add dpdkvhostuserclient0 to ovs bridge too.
      [root@dell-per750-37 ~]# ovs-vsctl show
      27e5be3e-c9e3-4941-8321-aeda5f851dd7
      Bridge ovsbr0
      datapath_type: netdev
      Port dpdk0
      Interface dpdk0
      type: dpdk
      options:

      {dpdk-devargs="0000:17:00.0"}

      Port ovsbr0
      Interface ovsbr0
      type: internal
      Port dpdkvhostuserclient0
      Interface dpdkvhostuserclient0
      type: dpdkvhostuserclient
      options:

      {vhost-server-path="/tmp/dpdkvhostuserclient0"}

      ovs_version: "2.17.7"

      2. configure mtu to 9000
      ovs-vsctl set Interface dpdkvhostuserclient0 mtu_request=9000
      ovs-vsctl set Interface dpdk0 mtu_request=9000

      2. Start guest with following xml
      [root@dell-per750-37 ~]# virsh dumpxml master
      <domain type='kvm' id='4'>
      <name>master</name>
      <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
      <memory unit='KiB'>5242880</memory>
      <currentMemory unit='KiB'>5242880</currentMemory>
      <memoryBacking>
      <hugepages>
      <page size='1048576' unit='KiB'/>
      </hugepages>
      <access mode='shared'/>
      </memoryBacking>
      <vcpu placement='static'>3</vcpu>
      <cputune>
      <vcpupin vcpu='2' cpuset='0'/>
      <emulatorpin cpuset='2'/>
      </cputune>
      <numatune>
      <memory mode='strict' nodeset='0'/>
      </numatune>
      <resource>
      <partition>/machine</partition>
      </resource>
      <os>
      <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
      <boot dev='hd'/>
      </os>
      <features>
      <acpi/>
      <apic/>
      <pmu state='off'/>
      <vmport state='off'/>
      <ioapic driver='qemu'/>
      </features>
      <cpu mode='host-passthrough' check='none' migratable='on'>
      <feature policy='require' name='tsc-deadline'/>
      <numa>
      <cell id='0' cpus='0-2' memory='5242880' unit='KiB' memAccess='shared'/>
      </numa>
      </cpu>
      <clock offset='utc'>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='hpet' present='no'/>
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>restart</on_crash>
      <pm>
      <suspend-to-mem enabled='no'/>
      <suspend-to-disk enabled='no'/>
      </pm>
      <devices>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/master.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </disk>
      <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
      </controller>
      <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
      </controller>
      <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </controller>
      <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </controller>
      <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </controller>
      <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </controller>
      <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </controller>
      <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      </controller>
      <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
      </controller>
      <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:e8'/>
      <source type='unix' path='/tmp/dpdkvhostuserclient0' mode='server'/>
      <target dev='dpdkvhostuserclient0'/>
      <model type='virtio'/>
      <driver name='vhost' iommu='on' ats='on'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </interface>
      <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet3'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </interface>
      <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
      <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
      </serial>
      <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target type='isa-serial' port='1'>
      <model name='isa-serial'/>
      </target>
      <alias name='serial1'/>
      </serial>
      <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      </console>
      <input type='mouse' bus='ps2'>
      <alias name='input0'/>
      </input>
      <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
      </input>
      <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
      </graphics>
      <audio id='1' type='none'/>
      <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </video>
      <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </memballoon>
      <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
      <alias name='iommu0'/>
      </iommu>
      </devices>
      <seclabel type='dynamic' model='selinux' relabel='yes'>
      <label>system_u:system_r:svirt_t:s0:c421,c512</label>
      <imagelabel>system_u:object_r:svirt_image_t:s0:c421,c512</imagelabel>
      </seclabel>
      <seclabel type='dynamic' model='dac' relabel='yes'>
      <label>+107:+1001</label>
      <imagelabel>+107:+1001</imagelabel>
      </seclabel>
      </domain>

      3. Configure the port inside guest
      ip address: 20.0.0.1
      mtu: 9000
      [root@localhost ~]# ip a
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
      2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
      link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff
      inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
      valid_lft 2682sec preferred_lft 2682sec
      inet6 fe80::bc2:f054:b47a:3336/64 scope link noprefixroute
      valid_lft forever preferred_lft forever
      3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
      link/ether 52:54:00:11:8f:e8 brd ff:ff:ff:ff:ff:ff
      inet 20.0.0.1/24 scope global enp3s0
      valid_lft forever preferred_lft forever
      inet6 fe80::43f0:cab:404f:8586/64 scope link noprefixroute
      valid_lft forever preferred_lft forever

      On client side:
      configure ip address and mtu 9000 on client:
      [root@dell-per740-55 ~]# ip a
      6: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
      link/ether f8:f2:1e:bb:cf:80 brd ff:ff:ff:ff:ff:ff
      altname enp94s0f0
      inet 20.0.0.3/24 scope global ens3f0
      valid_lft forever preferred_lft forever

      Inside server's guest ping client's port address:
      [root@localhost ~]# ping 20.0.0.3
      PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
      64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=0.060 ms
      64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.135 ms
      64 bytes from 20.0.0.3: icmp_seq=4 ttl=64 time=0.062 ms

      Start netserver on client side:
      [root@dell-per740-55 ~]# ps aux|grep netserver
      root 2144 0.0 0.0 3840 1352 ? Ss 03:25 0:00 netserver
      root 2291 0.0 0.0 6408 2300 pts/1 S+ 05:05 0:00 grep --color=auto netserver

      Do netperf test inside guest of server,
      [root@localhost ~]# netperf -t UDP_STREAM -H 20.0.0.3 – -m 1025
      MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 20.0.0.3 () port 0 AF_INET
      Socket Message Elapsed Messages
      Size Size Time Okay Errors Throughput
      bytes bytes secs # # 10^6bits/sec

      212992 1025 10.00 5205532 0 4268.52
      212992 10.00 5110575 4190.66

      [root@localhost ~]# netperf -t UDP_STREAM -H 20.0.0.3 – -m 8193
      MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 20.0.0.3 () port 0 AF_INET
      Socket Message Elapsed Messages
      Size Size Time Okay Errors Throughput
      bytes bytes secs # # 10^6bits/sec

      212992 8193 14.20 476933 0 2200.71
      212992 14.20 0 0.00

      Run ping again:
      [root@localhost ~]# ping 20.0.0.3
      PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
      ping: sendmsg: No buffer space available
      ping: sendmsg: No buffer space available

      Actual results:
      netperf test with 8193byte failed.
      And after do the netperf test with 8193byte, the guest cannot ping the client ip address anymore.

      Expected results:
      After do the netperf test with 8193byte, the guest should ping the client ip address successfully.
      When restart the guest, the guest still ping the client ip failed.
      [root@localhost ~]# ping 20.0.0.3
      PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
      From 20.0.0.1 icmp_seq=1 Destination Host Unreachable

      When restart the openvswitch service, the guest ping the client ip successfully.

      Additional info:
      https://beaker.engineering.redhat.com/jobs/7892906
      For another the i40e<->i40e test env, it also has this issue.

              mcoqueli@redhat.com Maxime Coquelin
              mcoqueli@redhat.com Maxime Coquelin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: