Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-110452

[Preliminary Testing Task][ipsec offload] crypto-offload : xfrmi : tcp performance drop

Linking RHIVOS CVEs to...Migration: Automation ...Sync from "Extern...XMLWordPrintable

    • rhel-net-core
    • 2
    • False
    • Hide

      None

      Show
      None
    • None

      What were you trying to do that didn't work?

      setup in6-out4 xfrmi, and ipsec is crypto offload,  tcp throughput drop 50%.  

      6.12.0-89.el10:  7.78 Gb/s

      6.12.0-109.el10: 3.49Gb/s

       

      |Distro    |kernel                 |driver    |topo|mtu | inv-outv| mode   |ARG            |crypto-OF:tcp| crypto-OF,udp|
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|1500 |in4-out4 |tunnel |rfc4106-gcm-aes |15.8 Gb/s    |3.45 Gb/s      |
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|1500 |in6-out4 |tunnel |rfc4106-gcm-aes |7.78 Gb/s    |2.84 Gb/s      |
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|1500 |in4-out6 |tunnel |rfc4106-gcm-aes |9.85 Gb/s    |3.27 Gb/s      |
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|1500 |in6-out6 |tunnel |rfc4106-gcm-aes |15 Gb/s      |3.23 Gb/s      |
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|9000 |in4-out4 |tunnel |rfc4106-gcm-aes |24.6 Gb/s    |14.7 Gb/s      |
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|9000 |in6-out4 |tunnel |rfc4106-gcm-aes |13.1 Gb/s    |13 Gb/s        |
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|9000 |in4-out6 |tunnel |rfc4106-gcm-aes |14.1 Gb/s    |12.9 Gb/s      |
      |RHEL-10.1 |6.12.0-89.el10.x86_64  |mlx5_core |nic|9000 |in6-out6 |tunnel |rfc4106-gcm-aes |24.5 Gb/s    |17.3 Gb/s      |
      |Distro    |kernel                 |driver    |topo|mtu | inv-outv| mode   |ARG            |crypto-OF:tcp| crypto-OF,udp|
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 1500|in4-out4| tunnel |rfc4106-gcm-aes |15.7Gb/s     |2.98Gb/s      |
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 1500|in6-out4| tunnel |rfc4106-gcm-aes |3.49Gb/s     |2.32Gb/s      |
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 1500|in4-out6| tunnel |rfc4106-gcm-aes |10Gb/s       |2.98Gb/s      |
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 1500|in6-out6| tunnel |rfc4106-gcm-aes |14.7Gb/s     |2.67Gb/s      |
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 9000|in4-out4| tunnel |rfc4106-gcm-aes |24.6Gb/s     |16.4Gb/s      |
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 9000|in6-out4| tunnel |rfc4106-gcm-aes |7.27Gb/s     |6.75Gb/s      |
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 9000|in4-out6| tunnel |rfc4106-gcm-aes |13.1Gb/s     |15Gb/s        |
      |RHEL-10.1 |6.12.0-109.el10.x86_64 |mlx5_core |nic| 9000|in6-out6| tunnel |rfc4106-gcm-aes |24.5Gb/s     |17Gb/s        |
      :: [ 11:55:29 ] :: [   PASS   ] :: Command 'create_env' (Expected 0, got 0)
      :: [ 11:55:29 ] :: [   LOG    ] :: Output of 'ethtool -i enp4s0f0np0':
      :: [ 11:55:29 ] :: [   LOG    ] :: --------------- OUTPUT START ---------------
      :: [ 11:55:29 ] :: [   LOG    ] :: driver: mlx5_core
      :: [ 11:55:29 ] :: [   LOG    ] :: version: 6.12.0-109.el10.x86_64
      :: [ 11:55:29 ] :: [   LOG    ] :: firmware-version: 22.42.1000 (MT_0000000536)
      :: [ 11:55:30 ] :: [   LOG    ] :: expansion-rom-version:
      :: [ 11:55:30 ] :: [   LOG    ] :: bus-info: 0000:04:00.0
      :: [ 11:55:30 ] :: [   LOG    ] :: supports-statistics: yes
      :: [ 11:55:30 ] :: [   LOG    ] :: supports-test: yes
      :: [ 11:55:30 ] :: [   LOG    ] :: supports-eeprom-access: no
      :: [ 11:55:30 ] :: [   LOG    ] :: supports-register-dump: no
      :: [ 11:55:30 ] :: [   LOG    ] :: supports-priv-flags: yes
      :: [ 11:55:30 ] :: [   LOG    ] :: ---------------  OUTPUT END  ---------------
      :: [ 11:55:30 ] :: [   PASS   ] :: Command 'ethtool -i enp4s0f0np0' (Expected 0, got 0)
      :: [ 11:55:30 ] :: [   LOG    ] :: Output of 'mstflint -d 04:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] q full':
      :: [ 11:55:30 ] :: [   LOG    ] :: --------------- OUTPUT START ---------------
      :: [ 11:55:30 ] :: [   LOG    ] :: Image type:            FS4
      :: [ 11:55:30 ] :: [   LOG    ] :: FW Version:            22.42.1000
      :: [ 11:55:30 ] :: [   LOG    ] :: FW Release Date:       8.8.2024
      :: [ 11:55:30 ] :: [   LOG    ] :: Part Number:           MCX621102AE-ADA_Ax
      :: [ 11:55:30 ] :: [   LOG    ] :: Description:           ConnectX-6 Dx EN adapter card; 25GbE ; Dual-port SFP28; PCIe 4.0 x8; Crypto; No Secure Boot;
      :: [ 11:55:30 ] :: [   LOG    ] :: Product Version:       22.42.1000
      :: [ 11:55:30 ] :: [   LOG    ] :: Rom Info:              type=UEFI version=14.35.15 cpu=AMD64,AARCH64
      :: [ 11:55:30 ] :: [   LOG    ] :: type=PXE version=3.7.500 cpu=AMD64
      :: [ 11:55:30 ] :: [   LOG    ] :: Description:           UID                GuidsNumber
      :: [ 11:55:30 ] :: [   LOG    ] :: Base GUID:             0c42a10300d81c24        4
      :: [ 11:55:30 ] :: [   LOG    ] :: Base MAC:              0c42a1d81c24            4
      :: [ 11:55:30 ] :: [   LOG    ] :: Image VSD:             N/A
      :: [ 11:55:30 ] :: [   LOG    ] :: Device VSD:            N/A
      :: [ 11:55:30 ] :: [   LOG    ] :: PSID:                  MT_0000000536
      :: [ 11:55:30 ] :: [   LOG    ] :: Security Attributes:   N/A
      :: [ 11:55:30 ] :: [   LOG    ] :: Default Update Method: fw_ctrl
      :: [ 11:55:30 ] :: [   LOG    ] :: Life cycle:            GA NON SECURED
      :: [ 11:55:30 ] :: [   LOG    ] :: Secure Boot Capable:   Enabled
      :: [ 11:55:30 ] :: [   LOG    ] :: ---------------  OUTPUT END  ---------------
      :: [ 11:55:30 ] :: [   PASS   ] :: Command 'mstflint -d 04:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] q full' (Expected 0, got 0)
      :: [ 11:55:30 ] :: [   PASS   ] :: Command 'ethtool -K enp4s0f0np0 gro on gso on tso on tx on rx on tx-esp-segmentation on esp-hw-offload on' (Expected 0-255, got 0)
      
      
      
      
      :: [ 12:01:17 ] :: [   LOG    ] :: #######################################
      :: [ 12:01:17 ] :: [   LOG    ] :: ##### crypto-offload tunnel in6:out4 #####
      :: [ 12:01:17 ] :: [   LOG    ] :: #######################################
      :: [ 12:01:17 ] :: [  BEGIN   ] :: Running 'setup_ipsec tunnel crypto-offload 6 4'
      ### set ipsec ###
      ++ ip xfrm state add src 5.5.40.2 dst 5.5.40.1 proto esp spi 1000 reqid 10000 aead 'rfc4106(gcm(aes))' 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128 mode tunnel sel src 1002:0::2 dst 1002:0::1 offload crypto dev enp4s0f0np0 dir out
      ++ ip xfrm state add src 5.5.40.1 dst 5.5.40.2 proto esp spi 1001 reqid 10001 aead 'rfc4106(gcm(aes))' 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128 mode tunnel sel src 1002:0::1 dst 1002:0::2 offload crypto dev enp4s0f0np0 dir in
      ++ ip xfrm policy add src 1002:0::2 dst 1002:0::1 dir out tmpl src 5.5.40.2 dst 5.5.40.1 proto esp reqid 10000 mode tunnel
      ++ ip xfrm policy add src 1002:0::1 dst 1002:0::2 dir in tmpl src 5.5.40.1 dst 5.5.40.2 proto esp reqid 10001 mode tunnel
      ++ ssh dell-per760-15.rhts.eng.pek2.redhat.com ip xfrm state add src 5.5.40.2 dst 5.5.40.1 proto esp spi 1000 reqid 10000 aead 'rfc4106\(gcm\(aes\)\)' 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128 mode tunnel sel src 1002:0::2 dst 1002:0::1 offload crypto dev ens5f0np0 dir in
      ++ ssh dell-per760-15.rhts.eng.pek2.redhat.com ip xfrm state add src 5.5.40.1 dst 5.5.40.2 proto esp spi 1001 reqid 10001 aead 'rfc4106\(gcm\(aes\)\)' 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128 mode tunnel sel src 1002:0::1 dst 1002:0::2 offload crypto dev ens5f0np0 dir out
      ++ ssh dell-per760-15.rhts.eng.pek2.redhat.com ip xfrm policy add src 1002:0::2 dst 1002:0::1 dir in tmpl src 5.5.40.2 dst 5.5.40.1 proto esp reqid 10000 mode tunnel
      ++ ssh dell-per760-15.rhts.eng.pek2.redhat.com ip xfrm policy add src 1002:0::1 dst 1002:0::2 dir out tmpl src 5.5.40.1 dst 5.5.40.2 proto esp reqid 10001 mode tunnel
      ++ set +x
      ********** ADDRESS AND XFRM  ON DUT *********
      8: enp4s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
          link/ether 0c:42:a1:d8:1c:24 brd ff:ff:ff:ff:ff:ff
          altname enx0c42a1d81c24
          inet 5.5.40.2/24 scope global enp4s0f0np0
             valid_lft forever preferred_lft forever
          inet6 1001::2/32 scope global 
             valid_lft forever preferred_lft forever
          inet6 fe80::e42:a1ff:fed8:1c24/64 scope link proto kernel_ll 
             valid_lft forever preferred_lft forever
      10: xfrmi0@enp4s0f0np0: <NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/none 
          inet 172.60.0.2/24 scope global xfrmi0
             valid_lft forever preferred_lft forever
          inet6 1002::2/32 scope global 
             valid_lft forever preferred_lft forever
          inet6 fe80::6ad1:cb26:ca7b:d876/64 scope link stable-privacy proto kernel_ll 
             valid_lft forever preferred_lft forever
      src 5.5.40.1 dst 5.5.40.2
      	proto esp spi 0x000003e9 reqid 10001 mode tunnel
      	replay-window 0 
      	aead rfc4106(gcm(aes)) 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128
      	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
      	crypto offload parameters: dev enp4s0f0np0 dir in mode crypto
      	dir in
      	sel src 1002::1/128 dst 1002::2/128 
      src 5.5.40.2 dst 5.5.40.1
      	proto esp spi 0x000003e8 reqid 10000 mode tunnel
      	replay-window 0 
      	aead rfc4106(gcm(aes)) 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128
      	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
      	crypto offload parameters: dev enp4s0f0np0 dir out mode crypto
      	dir out
      	sel src 1002::2/128 dst 1002::1/128 
      src 1002::1/128 dst 1002::2/128 
      	dir in priority 0 ptype main 
      	tmpl src 5.5.40.1 dst 5.5.40.2
      		proto esp reqid 10001 mode tunnel
      src 1002::2/128 dst 1002::1/128 
      	dir out priority 0 ptype main 
      	tmpl src 5.5.40.2 dst 5.5.40.1
      		proto esp reqid 10000 mode tunnel
      ********* ADDRESS AND XFRM  ON PEER ********
      6: ens5f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
          link/ether 0c:42:a1:d8:1c:30 brd ff:ff:ff:ff:ff:ff
          altname enp160s0f0np0
          altname enx0c42a1d81c30
          inet 5.5.40.1/24 scope global ens5f0np0
             valid_lft forever preferred_lft forever
          inet6 1001::1/32 scope global 
             valid_lft forever preferred_lft forever
          inet6 fe80::e42:a1ff:fed8:1c30/64 scope link proto kernel_ll 
             valid_lft forever preferred_lft forever
      8: xfrmi0@ens5f0np0: <NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/none 
          inet 172.60.0.1/24 scope global xfrmi0
             valid_lft forever preferred_lft forever
          inet6 1002::1/32 scope global 
             valid_lft forever preferred_lft forever
          inet6 fe80::72b4:626d:58fc:f7f9/64 scope link stable-privacy proto kernel_ll 
             valid_lft forever preferred_lft forever
      src 5.5.40.1 dst 5.5.40.2
      	proto esp spi 0x000003e9 reqid 10001 mode tunnel
      	replay-window 0 
      	aead rfc4106(gcm(aes)) 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128
      	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
      	crypto offload parameters: dev ens5f0np0 dir out mode crypto
      	dir out
      	sel src 1002::1/128 dst 1002::2/128 
      src 5.5.40.2 dst 5.5.40.1
      	proto esp spi 0x000003e8 reqid 10000 mode tunnel
      	replay-window 0 
      	aead rfc4106(gcm(aes)) 0x010203047aeaca3f87d060a12f4a4487d5a5c335 128
      	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
      	crypto offload parameters: dev ens5f0np0 dir in mode crypto
      	dir in
      	sel src 1002::2/128 dst 1002::1/128 
      src 1002::1/128 dst 1002::2/128 
      	dir out priority 0 ptype main 
      	tmpl src 5.5.40.1 dst 5.5.40.2
      		proto esp reqid 10001 mode tunnel
      src 1002::2/128 dst 1002::1/128 
      	dir in priority 0 ptype main 
      	tmpl src 5.5.40.2 dst 5.5.40.1
      		proto esp reqid 10000 mode tunnel
      :: [ 12:01:23 ] :: [   PASS   ] :: Command 'setup_ipsec tunnel crypto-offload 6 4' (Expected 0, got 0) 

      What is the impact of this issue to you?

      Please provide the package NVR for which the bug is seen:

      6.12.0-109.el10

      How reproducible is this bug?:

      always

      Steps to reproduce

      1.  
      2.  
      3.  

      Expected results

      Actual results

              MuXiumei Xiumei Mu
              MuXiumei Xiumei Mu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: