Description of problem:
/usr/sbin/devlink dev eswitch set pci 0000:06:00.0 mode switchdev

Version-Release number of selected component (if applicable):
kernel-core-5.14.0-0.rc6.46.el9.x86_64

How reproducible:
Most of the time

Steps to Reproduce:
1. Create eswitch and vhost devices
2. Reimage system (maybe just a reboot is sufficient here)
3. Re set mode switch dev on pci

Actual results:

Hung devlink command and a bunch of stack traces [1]

Expected results:
Should work

Additional info:

I was able to reproduce this on 2 servers, upgraded the firmware of the nic on one of them and it happened again.

Original firmware: 22.29.2002
Updated firmware: 22.31.1014

[1]
~~~
[Sun Sep 26 09:01:29 2021] device ovs-system entered promiscuous mode
[Sun Sep 26 09:01:29 2021] Timeout policy base is empty
[Sun Sep 26 09:01:29 2021] Failed to associated timeout policy `ovs_test_tp'
[Sun Sep 26 09:01:29 2021] device br-tenant entered promiscuous mode
[Sun Sep 26 09:01:31 2021] mlx5_core 0000:06:00.0 enp6s0f0np0: Link up
[Sun Sep 26 09:01:31 2021] IPv6: ADDRCONF(NETDEV_CHANGE): enp6s0f0np0: link becomes ready
[Sun Sep 26 09:01:33 2021] device enp6s0f0np0 entered promiscuous mode
[Sun Sep 26 09:01:33 2021] mlx5_core 0000:06:00.0 enp6s0f0np0: S-tagged traffic will be dropped while C-tag vlan stripping is enabled
[Sun Sep 26 09:01:34 2021] mlx5_core 0000:06:00.1 enp6s0f1np1: Link up
[Sun Sep 26 09:01:34 2021] IPv6: ADDRCONF(NETDEV_CHANGE): enp6s0f1np1: link becomes ready
[Sun Sep 26 09:01:36 2021] device enp6s0f1np1 entered promiscuous mode
[Sun Sep 26 09:01:37 2021] mlx5_core 0000:06:00.0: E-Switch: Supported tc chains and prios offload
[Sun Sep 26 09:01:37 2021] mlx5_core 0000:06:00.0: Supported tc offload range - chains: 4294967294, prios: 4294967295
[Sun Sep 26 09:01:37 2021] mlx5_core 0000:06:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[Sun Sep 26 09:04:39 2021] INFO: task kworker/u80:2:935 blocked for more than 122 seconds.
[Sun Sep 26 09:04:39 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:04:39 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:04:39 2021] task:kworker/u80:2 state stack: 0 pid: 935 ppid: 2 flags:0x00004000
[Sun Sep 26 09:04:39 2021] Workqueue: netns cleanup_net
[Sun Sep 26 09:04:39 2021] Call Trace:
[Sun Sep 26 09:04:39 2021] __schedule+0x206/0x550
[Sun Sep 26 09:04:39 2021] ? select_task_rq_fair+0x12e/0x360
[Sun Sep 26 09:04:39 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:04:39 2021] schedule_preempt_disabled+0xa/0x10
[Sun Sep 26 09:04:39 2021] __mutex_lock.constprop.0+0x29f/0x460
[Sun Sep 26 09:04:39 2021] ? idr_for_each+0x95/0xd0
[Sun Sep 26 09:04:39 2021] devlink_pernet_pre_exit+0x2a/0xc0
[Sun Sep 26 09:04:39 2021] cleanup_net+0x1d5/0x370
[Sun Sep 26 09:04:39 2021] process_one_work+0x1e6/0x380
[Sun Sep 26 09:04:39 2021] worker_thread+0x53/0x3d0
[Sun Sep 26 09:04:39 2021] ? process_one_work+0x380/0x380
[Sun Sep 26 09:04:39 2021] kthread+0x10f/0x130
[Sun Sep 26 09:04:39 2021] ? set_kthread_struct+0x40/0x40
[Sun Sep 26 09:04:39 2021] ret_from_fork+0x22/0x30
[Sun Sep 26 09:04:39 2021] INFO: task devlink:9205 blocked for more than 122 seconds.
[Sun Sep 26 09:04:39 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:04:39 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:04:39 2021] task:devlink state stack: 0 pid: 9205 ppid: 8492 flags:0x00004000
[Sun Sep 26 09:04:39 2021] Call Trace:
[Sun Sep 26 09:04:39 2021] __schedule+0x206/0x550
[Sun Sep 26 09:04:39 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:04:39 2021] rwsem_down_write_slowpath+0x22c/0x480
[Sun Sep 26 09:04:39 2021] register_netdevice_notifier+0x1c/0x110
[Sun Sep 26 09:04:39 2021] mlx5e_rep_bridge_init+0x111/0x130 [mlx5_core]
[Sun Sep 26 09:04:39 2021] mlx5e_uplink_rep_enable+0xd4/0x140 [mlx5_core]
[Sun Sep 26 09:04:39 2021] mlx5e_attach_netdev+0xa1/0x140 [mlx5_core]
[Sun Sep 26 09:04:39 2021] ? mlx5e_init_ul_rep+0x3e/0x50 [mlx5_core]
[Sun Sep 26 09:04:39 2021] mlx5e_netdev_attach_profile+0x93/0xb0 [mlx5_core]
[Sun Sep 26 09:04:39 2021] mlx5e_netdev_change_profile+0xa0/0xc0 [mlx5_core]
[Sun Sep 26 09:04:39 2021] mlx5e_vport_rep_load+0xa0/0xf0 [mlx5_core]
[Sun Sep 26 09:04:39 2021] mlx5_esw_offloads_rep_load+0x89/0xe0 [mlx5_core]
[Sun Sep 26 09:04:39 2021] esw_offloads_enable+0x266/0x370 [mlx5_core]
[Sun Sep 26 09:04:39 2021] mlx5_eswitch_enable_locked.part.0+0x100/0x310 [mlx5_core]
[Sun Sep 26 09:04:39 2021] esw_offloads_start+0x44/0x1f0 [mlx5_core]
[Sun Sep 26 09:04:39 2021] ? __nla_validate_parse+0x136/0x180
[Sun Sep 26 09:04:39 2021] mlx5_devlink_eswitch_mode_set+0x102/0x180 [mlx5_core]
[Sun Sep 26 09:04:39 2021] devlink_nl_cmd_eswitch_set_doit+0xc4/0x150
[Sun Sep 26 09:04:39 2021] genl_family_rcv_msg_doit+0xea/0x150
[Sun Sep 26 09:04:39 2021] genl_rcv_msg+0xdc/0x1e0
[Sun Sep 26 09:04:39 2021] ? __devlink_port_phys_port_name_get+0x1e0/0x1e0
[Sun Sep 26 09:04:39 2021] ? genl_get_cmd+0xd0/0xd0
[Sun Sep 26 09:04:39 2021] netlink_rcv_skb+0x50/0xf0
[Sun Sep 26 09:04:39 2021] genl_rcv+0x24/0x40
[Sun Sep 26 09:04:39 2021] netlink_unicast+0x201/0x2d0
[Sun Sep 26 09:04:39 2021] netlink_sendmsg+0x23b/0x480
[Sun Sep 26 09:04:39 2021] sock_sendmsg+0x5e/0x60
[Sun Sep 26 09:04:39 2021] __sys_sendto+0xf0/0x160
[Sun Sep 26 09:04:39 2021] ? handle_mm_fault+0xbf/0x280
[Sun Sep 26 09:04:39 2021] ? do_user_addr_fault+0x1c7/0x660
[Sun Sep 26 09:04:39 2021] __x64_sys_sendto+0x20/0x30
[Sun Sep 26 09:04:39 2021] do_syscall_64+0x3b/0x90
[Sun Sep 26 09:04:39 2021] entry_SYSCALL_64_after_hwframe+0x44/0xae
[Sun Sep 26 09:04:39 2021] RIP: 0033:0x7f4d9b9a71ea
[Sun Sep 26 09:04:39 2021] RSP: 002b:00007fffb5c9b488 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[Sun Sep 26 09:04:39 2021] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007f4d9b9a71ea
[Sun Sep 26 09:04:39 2021] RDX: 0000000000000038 RSI: 000055e2f7a85440 RDI: 0000000000000003
[Sun Sep 26 09:04:39 2021] RBP: 0000000000000000 R08: 00007f4d9baac200 R09: 000000000000000c
[Sun Sep 26 09:04:39 2021] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Sun Sep 26 09:04:39 2021] R13: 000055e2f7a852a0 R14: 000055e2f62cfd5c R15: 000055e2f7a85440
[Sun Sep 26 09:06:42 2021] INFO: task kworker/u80:2:935 blocked for more than 245 seconds.
[Sun Sep 26 09:06:42 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:06:42 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:06:42 2021] task:kworker/u80:2 state stack: 0 pid: 935 ppid: 2 flags:0x00004000
[Sun Sep 26 09:06:42 2021] Workqueue: netns cleanup_net
[Sun Sep 26 09:06:42 2021] Call Trace:
[Sun Sep 26 09:06:42 2021] __schedule+0x206/0x550
[Sun Sep 26 09:06:42 2021] ? select_task_rq_fair+0x12e/0x360
[Sun Sep 26 09:06:42 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:06:42 2021] schedule_preempt_disabled+0xa/0x10
[Sun Sep 26 09:06:42 2021] __mutex_lock.constprop.0+0x29f/0x460
[Sun Sep 26 09:06:42 2021] ? idr_for_each+0x95/0xd0
[Sun Sep 26 09:06:42 2021] devlink_pernet_pre_exit+0x2a/0xc0
[Sun Sep 26 09:06:42 2021] cleanup_net+0x1d5/0x370
[Sun Sep 26 09:06:42 2021] process_one_work+0x1e6/0x380
[Sun Sep 26 09:06:42 2021] worker_thread+0x53/0x3d0
[Sun Sep 26 09:06:42 2021] ? process_one_work+0x380/0x380
[Sun Sep 26 09:06:42 2021] kthread+0x10f/0x130
[Sun Sep 26 09:06:42 2021] ? set_kthread_struct+0x40/0x40
[Sun Sep 26 09:06:42 2021] ret_from_fork+0x22/0x30
[Sun Sep 26 09:06:42 2021] INFO: task devlink:9205 blocked for more than 245 seconds.
[Sun Sep 26 09:06:42 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:06:42 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:06:42 2021] task:devlink state stack: 0 pid: 9205 ppid: 1 flags:0x00004004
[Sun Sep 26 09:06:42 2021] Call Trace:
[Sun Sep 26 09:06:42 2021] __schedule+0x206/0x550
[Sun Sep 26 09:06:42 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:06:42 2021] rwsem_down_write_slowpath+0x22c/0x480
[Sun Sep 26 09:06:42 2021] register_netdevice_notifier+0x1c/0x110
[Sun Sep 26 09:06:42 2021] mlx5e_rep_bridge_init+0x111/0x130 [mlx5_core]
[Sun Sep 26 09:06:42 2021] mlx5e_uplink_rep_enable+0xd4/0x140 [mlx5_core]
[Sun Sep 26 09:06:42 2021] mlx5e_attach_netdev+0xa1/0x140 [mlx5_core]
[Sun Sep 26 09:06:42 2021] ? mlx5e_init_ul_rep+0x3e/0x50 [mlx5_core]
[Sun Sep 26 09:06:42 2021] mlx5e_netdev_attach_profile+0x93/0xb0 [mlx5_core]
[Sun Sep 26 09:06:42 2021] mlx5e_netdev_change_profile+0xa0/0xc0 [mlx5_core]
[Sun Sep 26 09:06:42 2021] mlx5e_vport_rep_load+0xa0/0xf0 [mlx5_core]
[Sun Sep 26 09:06:42 2021] mlx5_esw_offloads_rep_load+0x89/0xe0 [mlx5_core]
[Sun Sep 26 09:06:42 2021] esw_offloads_enable+0x266/0x370 [mlx5_core]
[Sun Sep 26 09:06:42 2021] mlx5_eswitch_enable_locked.part.0+0x100/0x310 [mlx5_core]
[Sun Sep 26 09:06:42 2021] esw_offloads_start+0x44/0x1f0 [mlx5_core]
[Sun Sep 26 09:06:42 2021] ? __nla_validate_parse+0x136/0x180
[Sun Sep 26 09:06:42 2021] mlx5_devlink_eswitch_mode_set+0x102/0x180 [mlx5_core]
[Sun Sep 26 09:06:42 2021] devlink_nl_cmd_eswitch_set_doit+0xc4/0x150
[Sun Sep 26 09:06:42 2021] genl_family_rcv_msg_doit+0xea/0x150
[Sun Sep 26 09:06:42 2021] genl_rcv_msg+0xdc/0x1e0
[Sun Sep 26 09:06:42 2021] ? __devlink_port_phys_port_name_get+0x1e0/0x1e0
[Sun Sep 26 09:06:42 2021] ? genl_get_cmd+0xd0/0xd0
[Sun Sep 26 09:06:42 2021] netlink_rcv_skb+0x50/0xf0
[Sun Sep 26 09:06:42 2021] genl_rcv+0x24/0x40
[Sun Sep 26 09:06:42 2021] netlink_unicast+0x201/0x2d0
[Sun Sep 26 09:06:42 2021] netlink_sendmsg+0x23b/0x480
[Sun Sep 26 09:06:42 2021] sock_sendmsg+0x5e/0x60
[Sun Sep 26 09:06:42 2021] __sys_sendto+0xf0/0x160
[Sun Sep 26 09:06:42 2021] ? handle_mm_fault+0xbf/0x280
[Sun Sep 26 09:06:42 2021] ? do_user_addr_fault+0x1c7/0x660
[Sun Sep 26 09:06:42 2021] __x64_sys_sendto+0x20/0x30
[Sun Sep 26 09:06:42 2021] do_syscall_64+0x3b/0x90
[Sun Sep 26 09:06:42 2021] entry_SYSCALL_64_after_hwframe+0x44/0xae
[Sun Sep 26 09:06:42 2021] RIP: 0033:0x7f4d9b9a71ea
[Sun Sep 26 09:06:42 2021] RSP: 002b:00007fffb5c9b488 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[Sun Sep 26 09:06:42 2021] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007f4d9b9a71ea
[Sun Sep 26 09:06:42 2021] RDX: 0000000000000038 RSI: 000055e2f7a85440 RDI: 0000000000000003
[Sun Sep 26 09:06:42 2021] RBP: 0000000000000000 R08: 00007f4d9baac200 R09: 000000000000000c
[Sun Sep 26 09:06:42 2021] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Sun Sep 26 09:06:42 2021] R13: 000055e2f7a852a0 R14: 000055e2f62cfd5c R15: 000055e2f7a85440
[Sun Sep 26 09:08:45 2021] INFO: task kworker/u80:2:935 blocked for more than 368 seconds.
[Sun Sep 26 09:08:45 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:08:45 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:08:45 2021] task:kworker/u80:2 state stack: 0 pid: 935 ppid: 2 flags:0x00004000
[Sun Sep 26 09:08:45 2021] Workqueue: netns cleanup_net
[Sun Sep 26 09:08:45 2021] Call Trace:
[Sun Sep 26 09:08:45 2021] __schedule+0x206/0x550
[Sun Sep 26 09:08:45 2021] ? select_task_rq_fair+0x12e/0x360
[Sun Sep 26 09:08:45 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:08:45 2021] schedule_preempt_disabled+0xa/0x10
[Sun Sep 26 09:08:45 2021] __mutex_lock.constprop.0+0x29f/0x460
[Sun Sep 26 09:08:45 2021] ? idr_for_each+0x95/0xd0
[Sun Sep 26 09:08:45 2021] devlink_pernet_pre_exit+0x2a/0xc0
[Sun Sep 26 09:08:45 2021] cleanup_net+0x1d5/0x370
[Sun Sep 26 09:08:45 2021] process_one_work+0x1e6/0x380
[Sun Sep 26 09:08:45 2021] worker_thread+0x53/0x3d0
[Sun Sep 26 09:08:45 2021] ? process_one_work+0x380/0x380
[Sun Sep 26 09:08:45 2021] kthread+0x10f/0x130
[Sun Sep 26 09:08:45 2021] ? set_kthread_struct+0x40/0x40
[Sun Sep 26 09:08:45 2021] ret_from_fork+0x22/0x30
[Sun Sep 26 09:08:45 2021] INFO: task devlink:9205 blocked for more than 368 seconds.
[Sun Sep 26 09:08:45 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:08:45 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:08:45 2021] task:devlink state stack: 0 pid: 9205 ppid: 1 flags:0x00004004
[Sun Sep 26 09:08:45 2021] Call Trace:
[Sun Sep 26 09:08:45 2021] __schedule+0x206/0x550
[Sun Sep 26 09:08:45 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:08:45 2021] rwsem_down_write_slowpath+0x22c/0x480
[Sun Sep 26 09:08:45 2021] register_netdevice_notifier+0x1c/0x110
[Sun Sep 26 09:08:45 2021] mlx5e_rep_bridge_init+0x111/0x130 [mlx5_core]
[Sun Sep 26 09:08:45 2021] mlx5e_uplink_rep_enable+0xd4/0x140 [mlx5_core]
[Sun Sep 26 09:08:45 2021] mlx5e_attach_netdev+0xa1/0x140 [mlx5_core]
[Sun Sep 26 09:08:45 2021] ? mlx5e_init_ul_rep+0x3e/0x50 [mlx5_core]
[Sun Sep 26 09:08:45 2021] mlx5e_netdev_attach_profile+0x93/0xb0 [mlx5_core]
[Sun Sep 26 09:08:45 2021] mlx5e_netdev_change_profile+0xa0/0xc0 [mlx5_core]
[Sun Sep 26 09:08:45 2021] mlx5e_vport_rep_load+0xa0/0xf0 [mlx5_core]
[Sun Sep 26 09:08:45 2021] mlx5_esw_offloads_rep_load+0x89/0xe0 [mlx5_core]
[Sun Sep 26 09:08:45 2021] esw_offloads_enable+0x266/0x370 [mlx5_core]
[Sun Sep 26 09:08:45 2021] mlx5_eswitch_enable_locked.part.0+0x100/0x310 [mlx5_core]
[Sun Sep 26 09:08:45 2021] esw_offloads_start+0x44/0x1f0 [mlx5_core]
[Sun Sep 26 09:08:45 2021] ? __nla_validate_parse+0x136/0x180
[Sun Sep 26 09:08:45 2021] mlx5_devlink_eswitch_mode_set+0x102/0x180 [mlx5_core]
[Sun Sep 26 09:08:45 2021] devlink_nl_cmd_eswitch_set_doit+0xc4/0x150
[Sun Sep 26 09:08:45 2021] genl_family_rcv_msg_doit+0xea/0x150
[Sun Sep 26 09:08:45 2021] genl_rcv_msg+0xdc/0x1e0
[Sun Sep 26 09:08:45 2021] ? __devlink_port_phys_port_name_get+0x1e0/0x1e0
[Sun Sep 26 09:08:45 2021] ? genl_get_cmd+0xd0/0xd0
[Sun Sep 26 09:08:45 2021] netlink_rcv_skb+0x50/0xf0
[Sun Sep 26 09:08:45 2021] genl_rcv+0x24/0x40
[Sun Sep 26 09:08:45 2021] netlink_unicast+0x201/0x2d0
[Sun Sep 26 09:08:45 2021] netlink_sendmsg+0x23b/0x480
[Sun Sep 26 09:08:45 2021] sock_sendmsg+0x5e/0x60
[Sun Sep 26 09:08:45 2021] __sys_sendto+0xf0/0x160
[Sun Sep 26 09:08:45 2021] ? handle_mm_fault+0xbf/0x280
[Sun Sep 26 09:08:45 2021] ? do_user_addr_fault+0x1c7/0x660
[Sun Sep 26 09:08:45 2021] __x64_sys_sendto+0x20/0x30
[Sun Sep 26 09:08:45 2021] do_syscall_64+0x3b/0x90
[Sun Sep 26 09:08:45 2021] entry_SYSCALL_64_after_hwframe+0x44/0xae
[Sun Sep 26 09:08:45 2021] RIP: 0033:0x7f4d9b9a71ea
[Sun Sep 26 09:08:45 2021] RSP: 002b:00007fffb5c9b488 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[Sun Sep 26 09:08:45 2021] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007f4d9b9a71ea
[Sun Sep 26 09:08:45 2021] RDX: 0000000000000038 RSI: 000055e2f7a85440 RDI: 0000000000000003
[Sun Sep 26 09:08:45 2021] RBP: 0000000000000000 R08: 00007f4d9baac200 R09: 000000000000000c
[Sun Sep 26 09:08:45 2021] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Sun Sep 26 09:08:45 2021] R13: 000055e2f7a852a0 R14: 000055e2f62cfd5c R15: 000055e2f7a85440
[Sun Sep 26 09:10:47 2021] INFO: task kworker/u80:2:935 blocked for more than 491 seconds.
[Sun Sep 26 09:10:47 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:10:47 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:10:47 2021] task:kworker/u80:2 state stack: 0 pid: 935 ppid: 2 flags:0x00004000
[Sun Sep 26 09:10:47 2021] Workqueue: netns cleanup_net
[Sun Sep 26 09:10:47 2021] Call Trace:
[Sun Sep 26 09:10:47 2021] __schedule+0x206/0x550
[Sun Sep 26 09:10:47 2021] ? select_task_rq_fair+0x12e/0x360
[Sun Sep 26 09:10:47 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:10:47 2021] schedule_preempt_disabled+0xa/0x10
[Sun Sep 26 09:10:47 2021] __mutex_lock.constprop.0+0x29f/0x460
[Sun Sep 26 09:10:48 2021] ? idr_for_each+0x95/0xd0
[Sun Sep 26 09:10:48 2021] devlink_pernet_pre_exit+0x2a/0xc0
[Sun Sep 26 09:10:48 2021] cleanup_net+0x1d5/0x370
[Sun Sep 26 09:10:48 2021] process_one_work+0x1e6/0x380
[Sun Sep 26 09:10:48 2021] worker_thread+0x53/0x3d0
[Sun Sep 26 09:10:48 2021] ? process_one_work+0x380/0x380
[Sun Sep 26 09:10:48 2021] kthread+0x10f/0x130
[Sun Sep 26 09:10:48 2021] ? set_kthread_struct+0x40/0x40
[Sun Sep 26 09:10:48 2021] ret_from_fork+0x22/0x30
[Sun Sep 26 09:10:48 2021] INFO: task devlink:9205 blocked for more than 491 seconds.
[Sun Sep 26 09:10:48 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:10:48 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:10:48 2021] task:devlink state stack: 0 pid: 9205 ppid: 1 flags:0x00004004
[Sun Sep 26 09:10:48 2021] Call Trace:
[Sun Sep 26 09:10:48 2021] __schedule+0x206/0x550
[Sun Sep 26 09:10:48 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:10:48 2021] rwsem_down_write_slowpath+0x22c/0x480
[Sun Sep 26 09:10:48 2021] register_netdevice_notifier+0x1c/0x110
[Sun Sep 26 09:10:48 2021] mlx5e_rep_bridge_init+0x111/0x130 [mlx5_core]
[Sun Sep 26 09:10:48 2021] mlx5e_uplink_rep_enable+0xd4/0x140 [mlx5_core]
[Sun Sep 26 09:10:48 2021] mlx5e_attach_netdev+0xa1/0x140 [mlx5_core]
[Sun Sep 26 09:10:48 2021] ? mlx5e_init_ul_rep+0x3e/0x50 [mlx5_core]
[Sun Sep 26 09:10:48 2021] mlx5e_netdev_attach_profile+0x93/0xb0 [mlx5_core]
[Sun Sep 26 09:10:48 2021] mlx5e_netdev_change_profile+0xa0/0xc0 [mlx5_core]
[Sun Sep 26 09:10:48 2021] mlx5e_vport_rep_load+0xa0/0xf0 [mlx5_core]
[Sun Sep 26 09:10:48 2021] mlx5_esw_offloads_rep_load+0x89/0xe0 [mlx5_core]
[Sun Sep 26 09:10:48 2021] esw_offloads_enable+0x266/0x370 [mlx5_core]
[Sun Sep 26 09:10:48 2021] mlx5_eswitch_enable_locked.part.0+0x100/0x310 [mlx5_core]
[Sun Sep 26 09:10:48 2021] esw_offloads_start+0x44/0x1f0 [mlx5_core]
[Sun Sep 26 09:10:48 2021] ? __nla_validate_parse+0x136/0x180
[Sun Sep 26 09:10:48 2021] mlx5_devlink_eswitch_mode_set+0x102/0x180 [mlx5_core]
[Sun Sep 26 09:10:48 2021] devlink_nl_cmd_eswitch_set_doit+0xc4/0x150
[Sun Sep 26 09:10:48 2021] genl_family_rcv_msg_doit+0xea/0x150
[Sun Sep 26 09:10:48 2021] genl_rcv_msg+0xdc/0x1e0
[Sun Sep 26 09:10:48 2021] ? __devlink_port_phys_port_name_get+0x1e0/0x1e0
[Sun Sep 26 09:10:48 2021] ? genl_get_cmd+0xd0/0xd0
[Sun Sep 26 09:10:48 2021] netlink_rcv_skb+0x50/0xf0
[Sun Sep 26 09:10:48 2021] genl_rcv+0x24/0x40
[Sun Sep 26 09:10:48 2021] netlink_unicast+0x201/0x2d0
[Sun Sep 26 09:10:48 2021] netlink_sendmsg+0x23b/0x480
[Sun Sep 26 09:10:48 2021] sock_sendmsg+0x5e/0x60
[Sun Sep 26 09:10:48 2021] __sys_sendto+0xf0/0x160
[Sun Sep 26 09:10:48 2021] ? handle_mm_fault+0xbf/0x280
[Sun Sep 26 09:10:48 2021] ? do_user_addr_fault+0x1c7/0x660
[Sun Sep 26 09:10:48 2021] __x64_sys_sendto+0x20/0x30
[Sun Sep 26 09:10:48 2021] do_syscall_64+0x3b/0x90
[Sun Sep 26 09:10:48 2021] entry_SYSCALL_64_after_hwframe+0x44/0xae
[Sun Sep 26 09:10:48 2021] RIP: 0033:0x7f4d9b9a71ea
[Sun Sep 26 09:10:48 2021] RSP: 002b:00007fffb5c9b488 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[Sun Sep 26 09:10:48 2021] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007f4d9b9a71ea
[Sun Sep 26 09:10:48 2021] RDX: 0000000000000038 RSI: 000055e2f7a85440 RDI: 0000000000000003
[Sun Sep 26 09:10:48 2021] RBP: 0000000000000000 R08: 00007f4d9baac200 R09: 000000000000000c
[Sun Sep 26 09:10:48 2021] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Sun Sep 26 09:10:48 2021] R13: 000055e2f7a852a0 R14: 000055e2f62cfd5c R15: 000055e2f7a85440
[Sun Sep 26 09:12:50 2021] INFO: task kworker/u80:2:935 blocked for more than 614 seconds.
[Sun Sep 26 09:12:50 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:12:50 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:12:50 2021] task:kworker/u80:2 state stack: 0 pid: 935 ppid: 2 flags:0x00004000
[Sun Sep 26 09:12:50 2021] Workqueue: netns cleanup_net
[Sun Sep 26 09:12:50 2021] Call Trace:
[Sun Sep 26 09:12:50 2021] __schedule+0x206/0x550
[Sun Sep 26 09:12:50 2021] ? select_task_rq_fair+0x12e/0x360
[Sun Sep 26 09:12:50 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:12:50 2021] schedule_preempt_disabled+0xa/0x10
[Sun Sep 26 09:12:50 2021] __mutex_lock.constprop.0+0x29f/0x460
[Sun Sep 26 09:12:50 2021] ? idr_for_each+0x95/0xd0
[Sun Sep 26 09:12:50 2021] devlink_pernet_pre_exit+0x2a/0xc0
[Sun Sep 26 09:12:50 2021] cleanup_net+0x1d5/0x370
[Sun Sep 26 09:12:50 2021] process_one_work+0x1e6/0x380
[Sun Sep 26 09:12:50 2021] worker_thread+0x53/0x3d0
[Sun Sep 26 09:12:50 2021] ? process_one_work+0x380/0x380
[Sun Sep 26 09:12:50 2021] kthread+0x10f/0x130
[Sun Sep 26 09:12:50 2021] ? set_kthread_struct+0x40/0x40
[Sun Sep 26 09:12:50 2021] ret_from_fork+0x22/0x30
[Sun Sep 26 09:12:50 2021] INFO: task devlink:9205 blocked for more than 614 seconds.
[Sun Sep 26 09:12:50 2021] Not tainted 5.14.0-0.rc6.46.el9.x86_64 #1
[Sun Sep 26 09:12:50 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Sep 26 09:12:50 2021] task:devlink state stack: 0 pid: 9205 ppid: 1 flags:0x00004004
[Sun Sep 26 09:12:50 2021] Call Trace:
[Sun Sep 26 09:12:50 2021] __schedule+0x206/0x550
[Sun Sep 26 09:12:50 2021] schedule+0x3c/0xa0
[Sun Sep 26 09:12:50 2021] rwsem_down_write_slowpath+0x22c/0x480
[Sun Sep 26 09:12:50 2021] register_netdevice_notifier+0x1c/0x110
[Sun Sep 26 09:12:50 2021] mlx5e_rep_bridge_init+0x111/0x130 [mlx5_core]
[Sun Sep 26 09:12:50 2021] mlx5e_uplink_rep_enable+0xd4/0x140 [mlx5_core]
[Sun Sep 26 09:12:50 2021] mlx5e_attach_netdev+0xa1/0x140 [mlx5_core]
[Sun Sep 26 09:12:50 2021] ? mlx5e_init_ul_rep+0x3e/0x50 [mlx5_core]
[Sun Sep 26 09:12:50 2021] mlx5e_netdev_attach_profile+0x93/0xb0 [mlx5_core]
[Sun Sep 26 09:12:51 2021] mlx5e_netdev_change_profile+0xa0/0xc0 [mlx5_core]
[Sun Sep 26 09:12:51 2021] mlx5e_vport_rep_load+0xa0/0xf0 [mlx5_core]
[Sun Sep 26 09:12:51 2021] mlx5_esw_offloads_rep_load+0x89/0xe0 [mlx5_core]
[Sun Sep 26 09:12:51 2021] esw_offloads_enable+0x266/0x370 [mlx5_core]
[Sun Sep 26 09:12:51 2021] mlx5_eswitch_enable_locked.part.0+0x100/0x310 [mlx5_core]
[Sun Sep 26 09:12:51 2021] esw_offloads_start+0x44/0x1f0 [mlx5_core]
[Sun Sep 26 09:12:51 2021] ? __nla_validate_parse+0x136/0x180
[Sun Sep 26 09:12:51 2021] mlx5_devlink_eswitch_mode_set+0x102/0x180 [mlx5_core]
[Sun Sep 26 09:12:51 2021] devlink_nl_cmd_eswitch_set_doit+0xc4/0x150
[Sun Sep 26 09:12:51 2021] genl_family_rcv_msg_doit+0xea/0x150
[Sun Sep 26 09:12:51 2021] genl_rcv_msg+0xdc/0x1e0
[Sun Sep 26 09:12:51 2021] ? __devlink_port_phys_port_name_get+0x1e0/0x1e0
[Sun Sep 26 09:12:51 2021] ? genl_get_cmd+0xd0/0xd0
[Sun Sep 26 09:12:51 2021] netlink_rcv_skb+0x50/0xf0
[Sun Sep 26 09:12:51 2021] genl_rcv+0x24/0x40
[Sun Sep 26 09:12:51 2021] netlink_unicast+0x201/0x2d0
[Sun Sep 26 09:12:51 2021] netlink_sendmsg+0x23b/0x480
[Sun Sep 26 09:12:51 2021] sock_sendmsg+0x5e/0x60
[Sun Sep 26 09:12:51 2021] __sys_sendto+0xf0/0x160
[Sun Sep 26 09:12:51 2021] ? handle_mm_fault+0xbf/0x280
[Sun Sep 26 09:12:51 2021] ? do_user_addr_fault+0x1c7/0x660
[Sun Sep 26 09:12:51 2021] __x64_sys_sendto+0x20/0x30
[Sun Sep 26 09:12:51 2021] do_syscall_64+0x3b/0x90
[Sun Sep 26 09:12:51 2021] entry_SYSCALL_64_after_hwframe+0x44/0xae
[Sun Sep 26 09:12:51 2021] RIP: 0033:0x7f4d9b9a71ea
[Sun Sep 26 09:12:51 2021] RSP: 002b:00007fffb5c9b488 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[Sun Sep 26 09:12:51 2021] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007f4d9b9a71ea
[Sun Sep 26 09:12:51 2021] RDX: 0000000000000038 RSI: 000055e2f7a85440 RDI: 0000000000000003
[Sun Sep 26 09:12:51 2021] RBP: 0000000000000000 R08: 00007f4d9baac200 R09: 000000000000000c
[Sun Sep 26 09:12:51 2021] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Sun Sep 26 09:12:51 2021] R13: 000055e2f7a852a0 R14: 000055e2f62cfd5c R15: 000055e2f7a85440
~~~