Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-2497

Test Coverage: mlx5_core driver: guest failed to ping each other when add vf port to ovs bridge

    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      ( ) The test coverage is aligned with the epic's acceptance criteria

      Given a system with OVS and mlx5_core driver managing VF ports,

      When VMs are connected to OVS using those VF ports and flows are added to OVS for forwarding traffic between them,

      Then the VMs should be able to communicate with each other successfully (e.g., ping commands between VMs should work).

      Show
      ( ) The test coverage is aligned with the epic's acceptance criteria Given a system with OVS and mlx5_core driver managing VF ports, When VMs are connected to OVS using those VF ports and flows are added to OVS for forwarding traffic between them, Then the VMs should be able to communicate with each other successfully (e.g., ping commands between VMs should work).
    • None
    • rhel-net-ovs-dpdk

      This task is tracking the test case writing activities to cover the bug described below.

      Description of problem:
      mlx5_core driver: guest failed to ping each other when add vf port to ovs bridge

      Version-Release number of selected component (if applicable):
      5.14.0-284.92.1.el9_2.x86_64
      openvswitch3.1-3.1.0-140.el9fdp
      openvswitch3.3-3.3.0-62.el9fdp
      openvswitch3.4-3.4.0-18.el9fdp

      How reproducible:
      Steps to Reproduce:
      echo 2 > /sys/devices/pci0000:5d/0000:5d:02.0/0000:5f:00.0/sriov_numvfs
      #configure the vf 0 mac same with the mac of enp0s9 port inside master4
      ip link set ens2f0np0 vf 0 mac 52:54:00:11:8f:e9
      #configure the vf 1 mac same with the mac of enp0s9 port inside master5
      ip link set ens2f0np0 vf 1 mac 52:54:00:11:8f:e8
      ovs-vsctl add-port ovsbr0 ens2f0v0
      ovs-vsctl add-port ovsbr0 ens2f0v1

      [root@dell-per740-03 ~]# ip link show
      8: ens2f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
      link/ether 94:6d:ae:b8:3f:a2 brd ff:ff:ff:ff:ff:ff
      vf 0 link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on, query_rss off
      vf 1 link/ether 52:54:00:11:8f:e8 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on, query_rss off
      altname enp95s0f0np0

      start guest master4 with following xml:

      [root@dell-per740-03 ~]# virsh dumpxml master4
      <domain type='kvm' id='1'>
        <name>master4</name>
        <uuid>37425e76-af6a-44a6-aba0-73434afe34c2</uuid>
        <memory unit='KiB'>8388608</memory>
        <currentMemory unit='KiB'>8388608</currentMemory>
        <memoryBacking>
          <hugepages>
            <page size='1048576' unit='KiB' nodeset='0'/>
          </hugepages>
          <access mode='shared'/>
        </memoryBacking>
        <vcpu placement='static'>3</vcpu>
        <cputune>
          <vcpupin vcpu='0' cpuset='0'/>
          <vcpupin vcpu='1' cpuset='1'/>
          <vcpupin vcpu='2' cpuset='2'/>
          <emulatorpin cpuset='1'/>
        </cputune>
        <resource>
          <partition>/machine</partition>
        </resource>
        <os>
          <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
          <boot dev='hd'/>
        </os>
        <features>
          <acpi/>
          <apic/>
        </features>
        <cpu mode='host-passthrough' check='none' migratable='on'>
          <feature policy='require' name='tsc-deadline'/>
          <numa>
            <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
          </numa>
        </cpu>
        <clock offset='utc'>
          <timer name='rtc' tickpolicy='catchup'/>
          <timer name='pit' tickpolicy='delay'/>
          <timer name='hpet' present='no'/>
        </clock>
        <on_poweroff>destroy</on_poweroff>
        <on_reboot>restart</on_reboot>
        <on_crash>restart</on_crash>
        <pm>
          <suspend-to-mem enabled='no'/>
          <suspend-to-disk enabled='no'/>
        </pm>
        <devices>
          <emulator>/usr/libexec/qemu-kvm</emulator>
          <disk type='file' device='disk'>
            <driver name='qemu' type='qcow2'/>
            <source file='/var/lib/libvirt/images/master4.qcow2' index='1'/>
            <backingStore/>
            <target dev='vda' bus='virtio'/>
            <alias name='virtio-disk0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
          </disk>
          <controller type='usb' index='0' model='ich9-ehci1'>
            <alias name='usb'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
          </controller>
          <controller type='usb' index='0' model='ich9-uhci1'>
            <alias name='usb'/>
            <master startport='0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
          </controller>
          <controller type='usb' index='0' model='ich9-uhci2'>
            <alias name='usb'/>
            <master startport='2'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
          </controller>
          <controller type='usb' index='0' model='ich9-uhci3'>
            <alias name='usb'/>
            <master startport='4'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
          </controller>
          <controller type='pci' index='0' model='pcie-root'>
            <alias name='pcie.0'/>
          </controller>
          <controller type='virtio-serial' index='0'>
            <alias name='virtio-serial0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
          </controller>
          <controller type='sata' index='0'>
            <alias name='ide'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
          </controller>
          <interface type='bridge'>
            <mac address='52:54:00:11:8f:e9'/>
            <source bridge='ovsbr0'/>
            <virtualport type='openvswitch'>
              <parameters interfaceid='f54fd445-25b0-4231-8ea7-b83f2a1f8df2'/>
            </virtualport>
            <target dev='tap0'/>
            <model type='virtio'/>
            <alias name='net0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
          </interface>
          <interface type='bridge'>
            <mac address='52:54:00:bb:63:7b'/>
            <source bridge='virbr0'/>
            <target dev='vnet0'/>
            <model type='virtio'/>
            <alias name='net1'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
          </interface>
          <serial type='pty'>
            <source path='/dev/pts/0'/>
            <target type='isa-serial' port='0'>
              <model name='isa-serial'/>
            </target>
            <alias name='serial0'/>
          </serial>
          <console type='pty' tty='/dev/pts/0'>
            <source path='/dev/pts/0'/>
            <target type='serial' port='0'/>
            <alias name='serial0'/>
          </console>
          <channel type='unix'>
            <source mode='bind' path='/run/libvirt/qemu/channel/1-master4/org.qemu.guest_agent.0'/>
            <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
            <alias name='channel0'/>
            <address type='virtio-serial' controller='0' bus='0' port='1'/>
          </channel>
          <input type='tablet' bus='usb'>
            <alias name='input0'/>
            <address type='usb' bus='0' port='1'/>
          </input>
          <input type='mouse' bus='ps2'>
            <alias name='input1'/>
          </input>
          <input type='keyboard' bus='ps2'>
            <alias name='input2'/>
          </input>
          <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
            <listen type='address' address='0.0.0.0'/>
          </graphics>
          <audio id='1' type='none'/>
          <video>
            <model type='virtio' vram='16384' heads='1' primary='yes'/>
            <alias name='video0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
          </video>
          <memballoon model='virtio'>
            <alias name='balloon0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
          </memballoon>
        </devices>
        <seclabel type='dynamic' model='selinux' relabel='yes'>
          <label>system_u:system_r:svirt_t:s0:c699,c734</label>
          <imagelabel>system_u:object_r:svirt_image_t:s0:c699,c734</imagelabel>
        </seclabel>
        <seclabel type='dynamic' model='dac' relabel='yes'>
          <label>+107:+981</label>
          <imagelabel>+107:+981</imagelabel>
        </seclabel>
      </domain>
      
      

      start guest master5 with following xml:

      [root@dell-per740-03 ~]# virsh dumpxml master5
      <domain type='kvm' id='2'>
        <name>master5</name>
        <uuid>37425e76-af6a-44a6-aba0-73434afe34c3</uuid>
        <memory unit='KiB'>8388608</memory>
        <currentMemory unit='KiB'>8388608</currentMemory>
        <memoryBacking>
          <hugepages>
            <page size='1048576' unit='KiB' nodeset='0'/>
          </hugepages>
          <access mode='shared'/>
        </memoryBacking>
        <vcpu placement='static'>3</vcpu>
        <cputune>
          <vcpupin vcpu='0' cpuset='3'/>
          <vcpupin vcpu='1' cpuset='4'/>
          <vcpupin vcpu='2' cpuset='5'/>
          <emulatorpin cpuset='6'/>
        </cputune>
        <resource>
          <partition>/machine</partition>
        </resource>
        <os>
          <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
          <boot dev='hd'/>
        </os>
        <features>
          <acpi/>
          <apic/>
        </features>
        <cpu mode='host-passthrough' check='none' migratable='on'>
          <feature policy='require' name='tsc-deadline'/>
          <numa>
            <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
          </numa>
        </cpu>
        <clock offset='utc'>
          <timer name='rtc' tickpolicy='catchup'/>
          <timer name='pit' tickpolicy='delay'/>
          <timer name='hpet' present='no'/>
        </clock>
        <on_poweroff>destroy</on_poweroff>
        <on_reboot>restart</on_reboot>
        <on_crash>restart</on_crash>
        <pm>
          <suspend-to-mem enabled='no'/>
          <suspend-to-disk enabled='no'/>
        </pm>
        <devices>
          <emulator>/usr/libexec/qemu-kvm</emulator>
          <disk type='file' device='disk'>
            <driver name='qemu' type='qcow2'/>
            <source file='/var/lib/libvirt/images/master5.qcow2' index='1'/>
            <backingStore/>
            <target dev='vda' bus='virtio'/>
            <alias name='virtio-disk0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
          </disk>
          <controller type='usb' index='0' model='ich9-ehci1'>
            <alias name='usb'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
          </controller>
          <controller type='usb' index='0' model='ich9-uhci1'>
            <alias name='usb'/>
            <master startport='0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
          </controller>
          <controller type='usb' index='0' model='ich9-uhci2'>
            <alias name='usb'/>
            <master startport='2'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
          </controller>
          <controller type='usb' index='0' model='ich9-uhci3'>
            <alias name='usb'/>
            <master startport='4'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
          </controller>
          <controller type='pci' index='0' model='pcie-root'>
            <alias name='pcie.0'/>
          </controller>
          <controller type='virtio-serial' index='0'>
            <alias name='virtio-serial0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
          </controller>
          <controller type='sata' index='0'>
            <alias name='ide'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
          </controller>
          <interface type='bridge'>
            <mac address='52:54:00:11:8f:e8'/>
            <source bridge='ovsbr0'/>
            <virtualport type='openvswitch'>
              <parameters interfaceid='f54fd445-25b0-4231-8ea7-b83f2a1f8df2'/>
            </virtualport>
            <target dev='tap1'/>
            <model type='virtio'/>
            <alias name='net0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
          </interface>
          <interface type='bridge'>
            <mac address='52:54:00:bb:63:7a'/>
            <source bridge='virbr0'/>
            <target dev='vnet1'/>
            <model type='virtio'/>
            <alias name='net1'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
          </interface>
          <serial type='pty'>
            <source path='/dev/pts/1'/>
            <target type='isa-serial' port='0'>
              <model name='isa-serial'/>
            </target>
            <alias name='serial0'/>
          </serial>
          <console type='pty' tty='/dev/pts/1'>
            <source path='/dev/pts/1'/>
            <target type='serial' port='0'/>
            <alias name='serial0'/>
          </console>
          <channel type='unix'>
            <source mode='bind' path='/run/libvirt/qemu/channel/2-master5/org.qemu.guest_agent.0'/>
            <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
            <alias name='channel0'/>
            <address type='virtio-serial' controller='0' bus='0' port='1'/>
          </channel>
          <input type='tablet' bus='usb'>
            <alias name='input0'/>
            <address type='usb' bus='0' port='1'/>
          </input>
          <input type='mouse' bus='ps2'>
            <alias name='input1'/>
          </input>
          <input type='keyboard' bus='ps2'>
            <alias name='input2'/>
          </input>
          <graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0'>
            <listen type='address' address='0.0.0.0'/>
          </graphics>
          <audio id='1' type='none'/>
          <video>
            <model type='virtio' vram='16384' heads='1' primary='yes'/>
            <alias name='video0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
          </video>
          <memballoon model='virtio'>
            <alias name='balloon0'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
          </memballoon>
        </devices>
        <seclabel type='dynamic' model='selinux' relabel='yes'>
          <label>system_u:system_r:svirt_t:s0:c338,c552</label>
          <imagelabel>system_u:object_r:svirt_image_t:s0:c338,c552</imagelabel>
        </seclabel>
        <seclabel type='dynamic' model='dac' relabel='yes'>
          <label>+107:+981</label>
          <imagelabel>+107:+981</imagelabel>
        </seclabel>
      </domain>
      
      

      ovs topo as following:

      [root@dell-per740-03 ~]# ovs-vsctl show
      1d256f50-1961-4f71-875d-5a7f2d7109a5
          Bridge ovsbr0
              Port ens2f0v0
                  Interface ens2f0v0
              Port tap0
                  Interface tap0
              Port ens2f0v1
                  Interface ens2f0v1
              Port ovsbr0
                  Interface ovsbr0
                      type: internal
              Port tap1
                  Interface tap1
          ovs_version: "3.1.7-140.el9fdp"
      

      ovs-ofctl add-flow ovsbr0 in_port=ens2f0v0,actions=output:tap0
      ovs-ofctl add-flow ovsbr0 in_port=ens2f0v1,actions=output:tap1
      ovs-ofctl add-flow ovsbr0 in_port=tap0,actions=output:ens2f0v0
      ovs-ofctl add-flow ovsbr0 in_port=tap1,actions=output:ens2f0v1

      configure the ip address 20.0.0.2/24 to the port inside master4

      [root@dell-per740-03 ~]# virsh console master4
      Connected to domain 'master4'
      Escape character is ^] (Ctrl + ])
      
      [root@localhost ~]# ip a
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
             valid_lft forever preferred_lft forever
          inet6 ::1/128 scope host 
             valid_lft forever preferred_lft forever
      2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
          link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff
          inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp0s3
             valid_lft 2855sec preferred_lft 2855sec
          inet6 fe80::cf73:83d6:70a8:8087/64 scope link noprefixroute 
             valid_lft forever preferred_lft forever
      3: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
          link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff
          inet 20.0.0.2/24 scope global enp0s9
             valid_lft forever preferred_lft forever
          inet6 2001:5c0:9168::2/64 scope global 
             valid_lft forever preferred_lft forever
          inet6 fe80::604:b110:7ceb:4386/64 scope link noprefixroute 
             valid_lft forever preferred_lft forever
      
      

      configure the ip address 20.0.0.3/24 to the port inside master5

      [root@dell-per740-03 ~]# virsh console master5
      Connected to domain 'master5'
      Escape character is ^] (Ctrl + ])
      
      [root@localhost ~]# ip a
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
             valid_lft forever preferred_lft forever
          inet6 ::1/128 scope host 
             valid_lft forever preferred_lft forever
      2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
          link/ether 52:54:00:bb:63:7a brd ff:ff:ff:ff:ff:ff
          inet 192.168.122.137/24 brd 192.168.122.255 scope global dynamic noprefixroute enp0s3
             valid_lft 2884sec preferred_lft 2884sec
          inet6 fe80::b65c:5598:50ed:37ac/64 scope link noprefixroute 
             valid_lft forever preferred_lft forever
      3: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
          link/ether 52:54:00:11:8f:e8 brd ff:ff:ff:ff:ff:ff
          inet 20.0.0.3/24 scope global enp0s9
             valid_lft forever preferred_lft forever
          inet6 2001:5c0:9168::3/64 scope global 
             valid_lft forever preferred_lft forever
      
      

      Inside master4, ping the ip address(20.0.0.3) of master5:

      Actual results:
      guest failed to ping each other when add vf port to ovs bridge
      [root@localhost ~]# ping 20.0.0.3
      PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
      From 20.0.0.2 icmp_seq=1 Destination Host Unreachable
      From 20.0.0.2 icmp_seq=5 Destination Host Unreachable
      From 20.0.0.2 icmp_seq=6 Destination Host Unreachable

      Expected results:
      guest ping each other successfully when add vf port to ovs bridge

      Additional info:

              ovsdpdk-triage ovsdpdk triage
              nstbot NST Bot
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: