Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-13471

Infiniband/mlx5_ib stack adds a big toll to netns creation time

Linking RHIVOS CVEs to...Migration: Automation ...SWIFT: POC ConversionSync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • rhel-8.8.0
    • kernel / RDMA
    • None
    • Important
    • 1
    • rhel-net-drivers
    • 1
    • False
    • False
    • Hide

      None

      Show
      None
    • None
    • Network Driver 4
    • None
    • None
    • If docs needed, set a value
    • None
    • 57,005

      Description of problem:
      Having the infiniband stack and perhaps more especificly mlx5_ib driver loaded causes a ~20% slowdown in netns creation:

      [root@wsfd-netdev89 ~]# time for i in

      {1..1000}; do ip netns add a; ip netns del a ; done

      real 0m23,309s
      user 0m1,241s
      sys 0m4,192s
      [root@wsfd-netdev89 ~]# rmmod mlx5_ib
      [root@wsfd-netdev89 ~]# time for i in {1..1000}

      ; do ip netns add a; ip netns del a ; done

      real 0m19,939s
      user 0m0,830s
      sys 0m2,807s

      then I unloaded all IB modules and:
      [root@wsfd-netdev89 ~]# time for i in

      {1..1000}

      ; do ip netns add a; ip netns del a ; done

      real 0m19,818s
      user 0m0,770s
      sys 0m2,862s

      These tests done while dumping tc filters with stats in parallel.

      Perf record gives:

      • 67,11% 0,16% ip [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe◆
        66,95% entry_SYSCALL_64_after_hwframe ▒
      • do_syscall_64 ▒
      • 56,44% __x64_sys_unshare ▒
        ksys_unshare ▒
        unshare_nsproxy_namespaces ▒
        create_new_namespaces ▒
        copy_net_ns ▒
        setup_net ▒
      • ops_init ▒
      • 32,50% rdma_dev_init_net ▒
        ^^^^^^
      • add_one_compat_dev.part.25 ▒
      • 30,25% ib_setup_port_attrs ▒
      • 24,12% internal_create_groups.part.4 ▒
      • internal_create_group ▒
      • 23,51% sysfs_add_file_mode_ns ▒
      • __kernfs_create_file ▒
      • 18,02% kernfs_new_node ▒
      • 17,80% __kernfs_new_node ▒
        + 10,41% kmem_cache_alloc ▒
        + 2,70% idr_alloc_cyclic ▒
        + 2,05% kstrdup ▒
        + 1,44% security_kernfs_init_security ▒
        + 4,52% kernfs_add_one ▒
        + 2,74% alloc_port_table_group.isra.3 ▒
      • 1,46% ib_query_port ▒
        + 0,81% mlx5_ib_query_port ▒
        hw getting queried --------^^^^^^
        + 0,64% mlx5_query_port_roce ▒
        + 1,07% kobject_add ▒
        + 2,25% device_add ▒
        + 8,31% ip_tunnel_init_net ▒
        + 2,70% tcp_sk_init ▒
        + 2,42% loopback_net_init ▒
        + 1,91% inet6_net_init ▒
        + 1,49% icmpv6_sk_init ▒
        + 1,25% icmp_sk_init ▒
        + 1,08% nf_conntrack_pernet_init ▒
        + 1,04% ipv4_mib_init_net ▒
        0,66% sunrpc_init_net ▒
        + 2,02% __x64_sys_mount ▒
        + 1,86% __x64_sys_exit_group ▒
        + 1,77% ksys_mmap_pgoff ▒
        + 1,34% __x64_sys_mprotect ▒
        + 1,27% __x64_sys_execve ▒

      Version-Release number of selected component (if applicable):
      4.18.0-414.el8.x86_64

      How reproducible:
      Always

      Steps to Reproduce:
      1.blacklist mlx5_ib
      2.reboot
      3.test how long it takes to create 1000 netns
      4.load mlx5_ib
      5.test how long it takes to create 1000 netns again

      Actual results:
      Considerable slowness detected, about 20%.

      Expected results:
      Less of a impact, for a subsystem that is not being used on this host.

      Additional info:

              atzin AMIR TZIN (Inactive)
              rhn-support-mleitner Marcelo Leitner
              NVIDIA (Mellanox) Confidential Group
              AMIR TZIN AMIR TZIN (Inactive)
              Afom Michael Afom Michael
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: