[ 6.157110] systemd[1]: Mounting Temporary Directory /tmp... [ 6.158909] systemd[1]: Starting CoreOS: Set printk To Level 4 (warn)... [ 6.159447] systemd[1]: Ignition (delete config) was skipped because of an unmet condition check (ConditionFirstBoot=true). [ 6.161066] systemd[1]: Starting Create List of Static Device Nodes... [ 6.162776] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... [ 6.164587] systemd[1]: Starting Load Kernel Module configfs... [ 6.166141] systemd[1]: Starting Load Kernel Module drm... [ 6.167821] systemd[1]: Starting Load Kernel Module efi_pstore... [ 6.169495] systemd[1]: Starting Load Kernel Module fuse... [ 6.169922] systemd[1]: ostree-prepare-root.service: Deactivated successfully. [ 6.170103] systemd[1]: Stopped OSTree Prepare OS/. [ 6.170572] systemd[1]: systemd-fsck-root.service: Deactivated successfully. [ 6.170715] systemd[1]: Stopped File System Check on Root Device. [ 6.171103] systemd[1]: Stopped Journal Service. [ 6.173172] systemd[1]: Starting Journal Service... [ 6.175266] systemd[1]: Starting Load Kernel Modules... [ 6.176881] systemd[1]: Starting Generate network units from Kernel command line... [ 6.177285] systemd[1]: TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f). [ 6.178950] systemd[1]: Starting Remount Root and Kernel File Systems... [ 6.180704] systemd[1]: Starting Coldplug All udev Devices... [ 6.183293] systemd[1]: Mounted Huge Pages File System. [ 6.183718] systemd[1]: Mounted POSIX Message Queue File System. [ 6.184118] systemd[1]: Mounted Kernel Debug File System. [ 6.184504] systemd[1]: Mounted Kernel Trace File System. [ 6.184880] systemd[1]: Mounted Temporary Directory /tmp. [ 6.185584] systemd[1]: Finished CoreOS: Set printk To Level 4 (warn). [ 6.186298] systemd[1]: Finished Create List of Static Device Nodes. [ 6.186820] systemd[1]: modprobe@configfs.service: Deactivated successfully. [ 6.187050] systemd[1]: Finished Load Kernel Module configfs. [ 6.187601] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully. [ 6.187851] systemd[1]: Finished Load Kernel Module efi_pstore. [ 6.188372] systemd[1]: modprobe@fuse.service: Deactivated successfully. [ 6.188592] systemd[1]: Finished Load Kernel Module fuse. [ 6.189073] systemd[1]: systemd-network-generator.service: Main process exited, code=exited, status=1/FAILURE [ 6.189305] systemd[1]: systemd-network-generator.service: Failed with result 'exit-code'. [ 6.189549] systemd[1]: Failed to start Generate network units from Kernel command line. [ 6.190396] systemd[1]: Finished Remount Root and Kernel File Systems. [ 6.192207] systemd[1]: Mounting FUSE Control File System... [ 6.193810] systemd[1]: Mounting Kernel Configuration File System... [ 6.194215] systemd[1]: Special handling of early boot iSCSI sessions was skipped because of an unmet condition check (ConditionDirectoryNotEmpty=/sys/class/iscsi_session). [ 6.197171] systemd[1]: Rebuild Hardware Database was skipped because of an unmet condition check (ConditionNeedsUpdate=/etc). [ 6.197395] systemd[1]: Create System Users was skipped because no trigger condition checks were met. [ 6.198885] systemd[1]: Starting Create Static Device Nodes in /dev... [ 6.198902] Warning: Unmaintained driver is detected: ip_tables [ 6.200479] systemd[1]: Mounted FUSE Control File System. [ 6.201115] systemd[1]: Finished Load Kernel Modules. [ 6.201577] systemd[1]: Mounted Kernel Configuration File System. [ 6.203443] systemd[1]: Starting Apply Kernel Variables... [ 6.207742] systemd[1]: Started Journal Service. [ 6.249724] ACPI: bus type drm_connector registered [ 6.403452] IPMI message handler: version 39.2 [ 6.407401] ipmi device interface [ 6.417786] ipmi_si: IPMI System Interface driver [ 6.417842] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS [ 6.417847] ipmi_platform: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 [ 6.417852] ipmi_si: Adding SMBIOS-specified kcs state machine [ 6.418185] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI [ 6.418201] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.418205] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.418246] ipmi_si IPI0001:00: ipmi_platform: [io 0x0ca8] regsize 1 spacing 4 irq 10 [ 6.418773] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.418776] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.419015] ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI [ 6.419019] ipmi_si: Adding ACPI-specified kcs state machine [ 6.419145] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.419149] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.419165] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 [ 6.419942] lpc_ich 0000:00:1f.0: I/O space for ACPI uninitialized [ 6.419949] lpc_ich 0000:00:1f.0: No MFD cells added [ 6.447804] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.447811] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.449102] usbcore: registered new interface driver cp210x [ 6.449155] usbserial: USB Serial support registered for cp210x [ 6.449183] cp210x 1-13:1.0: cp210x converter detected [ 6.449259] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 6.449504] i801_smbus 0000:00:1f.4: SPD Write Disable is set [ 6.449540] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt [ 6.449801] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.449804] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.451638] usb 1-13: cp210x converter now attached to ttyUSB0 [ 6.451680] cp210x 1-13:1.1: cp210x converter detected [ 6.451902] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.451906] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.452984] cp210x 1-13:1.1: Failed to attach ACPI GPIO chip [ 6.453635] usb 1-13: cp210x converter now attached to ttyUSB1 [ 6.453684] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.453689] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.453909] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.453911] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.465261] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.4) [ 6.467159] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.467164] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.467422] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.467426] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.467734] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.467738] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.468070] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.468072] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.471106] i2c i2c-2: 12/24 memory slots populated (from DMI) [ 6.471109] i2c i2c-2: Systems with more than 8 memory slots not supported yet, not instantiating SPD [ 6.471387] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.471397] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.474909] iTCO_vendor_support: vendor-support=0 [ 6.477413] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.477417] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.477620] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.477623] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.482006] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ 6.482020] iTCO_wdt iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400) [ 6.482318] iTCO_wdt iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) [ 6.482454] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.482458] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.482489] RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 655360 ms ovfl timer [ 6.482493] RAPL PMU: hw unit of domain package 2^-14 Joules [ 6.482495] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 6.488292] mgag200 0000:03:00.0: vgaarb: deactivate vga console [ 6.493662] Console: switching to colour dummy device 80x25 [ 6.498405] [drm] Initialized mgag200 1.0.0 for 0000:03:00.0 on minor 0 [ 6.503755] fbcon: mgag200drmfb (fb0) is primary device [ 6.554158] Console: switching to colour frame buffer device 128x48 [ 6.586332] mgag200 0000:03:00.0: [drm] fb0: mgag200drmfb frame buffer device [ 6.668377] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.668385] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.755609] EDAC MC0: Giving out device to module skx_edac controller Skylake Socket#0 IMC#0: DEV 0000:3a:0a.0 (INTERRUPT) [ 6.755654] EDAC MC1: Giving out device to module skx_edac controller Skylake Socket#0 IMC#1: DEV 0000:3a:0c.0 (INTERRUPT) [ 6.755697] EDAC MC2: Giving out device to module skx_edac controller Skylake Socket#1 IMC#0: DEV 0000:ae:0a.0 (INTERRUPT) [ 6.755738] EDAC MC3: Giving out device to module skx_edac controller Skylake Socket#1 IMC#1: DEV 0000:ae:0c.0 (INTERRUPT) [ 6.793557] intel_rapl_common: Found RAPL domain package [ 6.793567] intel_rapl_common: Found RAPL domain dram [ 6.793573] intel_rapl_common: package-0:dram:long_term locked by BIOS [ 6.793667] intel_rapl_common: Found RAPL domain package [ 6.793677] intel_rapl_common: Found RAPL domain dram [ 6.793683] intel_rapl_common: package-1:dram:long_term locked by BIOS [ 6.794286] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.794290] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.834094] ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. [ 6.852133] ipmi_si IPI0001:00: Using irq 10 [ 6.875052] ipmi_si IPI0001:00: IPMI message handler: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) [ 6.875329] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.875338] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 6.912636] ipmi_si IPI0001:00: IPMI kcs interface initialized [ 6.916940] ipmi_ssif: IPMI SSIF Interface driver [ 6.921706] power_meter ACPI000D:00: Found ACPI power meter. [ 6.921737] power_meter ACPI000D:00: hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info(). [ 6.921894] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 6.921897] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 7.524325] systemd-journald[1345]: Received client request to flush runtime journal. [ 7.528006] systemd-journald[1345]: File /var/log/journal/d501eda945e04be8a6d6585ab0fc31b9/system.journal corrupted or uncleanly shut down, renaming and replacing. [ 7.660453] EXT4-fs (sdb3): mounted filesystem 61a8b283-c58a-4315-8205-61050f5d25cd ro with ordered data mode. Quota mode: none. [ 8.594830] openvswitch: Open vSwitch switching datapath [ 8.635970] ovs-system: entered promiscuous mode [ 8.637058] Timeout policy base is empty [ 8.652573] i40e 0000:18:00.0 eno1: entered promiscuous mode [ 8.656087] br-ex: entered promiscuous mode [ 8.659294] ovn-k8s-mp0: entered promiscuous mode [ 8.669586] genev_sys_6081: entered promiscuous mode [ 8.671203] br-int: entered promiscuous mode [ 9.435880] mlx5_core 0000:3b:00.0 ens1f0np0: Link down [ 9.859684] mlx5_core 0000:3b:00.1 ens1f1np1: Link down [ 9.885875] br-ex: left promiscuous mode [ 9.933095] i40e 0000:18:00.0 eno1: left promiscuous mode [ 12.291312] evm: overlay not supported [ 12.708063] igb 0000:01:00.0 eno3: igb: eno3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 14.305936] igb 0000:01:00.1 eno4: igb: eno4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 16.981644] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 16.981650] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 16.981719] pci 0000:87:00.0: deferred probe pending [ 70.425605] zram_generator::config[2677]: No configuration found. [ 70.426122] systemd-rc-local-generator[2666]: /etc/rc.d/rc.local is not marked executable, skipping. [ 70.428871] Skipping bootloader-migrate-generator: sysroot.bootloader is set [ 73.149637] i40e 0000:18:00.0 eno1: entered promiscuous mode [ 73.541582] i40e 0000:18:00.0 eno1: left promiscuous mode [ 73.542917] i40e 0000:18:00.0 eno1: entered promiscuous mode [ 74.392263] br-ex: entered promiscuous mode [ 76.573512] Warning: Unmaintained driver is detected: nft_compat [ 82.966612] octeon_ep: Unloading ... [ 82.984138] octeon_ep: Unloading complete [ 83.763932] zram_generator::config[5478]: No configuration found. [ 83.764608] systemd-rc-local-generator[5468]: /etc/rc.d/rc.local is not marked executable, skipping. [ 83.772117] Skipping bootloader-migrate-generator: sysroot.bootloader is set [ 85.385444] EXT4-fs (sdb3): re-mounted 61a8b283-c58a-4315-8205-61050f5d25cd r/w. Quota mode: none. [ 91.981476] a67741b5781c7c2: entered promiscuous mode [ 91.989803] be0eb060ec929cc: entered promiscuous mode [ 91.990144] 90d5f6cc2c4cf4a: entered promiscuous mode [ 92.004052] b0ac4d88f3b9ac3: entered promiscuous mode [ 92.004420] ca5dc55394f3984: entered promiscuous mode [ 103.039689] octeon_ep: Loading Marvell Octeon EndPoint NIC Driver ... [ 103.040161] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 103.040166] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 103.040316] octeon_ep: Loaded successfully ! [ 103.050284] octeon_ep_vf: Loading Marvell Octeon EndPoint NIC VF Driver ... [ 108.130220] octeon_ep_vf: Unloading ... [ 108.138909] octeon_ep_vf: Unloading complete [ 108.155710] octeon_ep: Unloading ... [ 108.166930] octeon_ep: Unloading complete [ 128.188496] octeon_ep: Loading Marvell Octeon EndPoint NIC Driver ... [ 128.188832] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 128.188838] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 128.188930] octeon_ep: Loaded successfully ! [ 128.198651] octeon_ep_vf: Loading Marvell Octeon EndPoint NIC VF Driver ... [ 133.275620] octeon_ep_vf: Unloading ... [ 133.283695] octeon_ep_vf: Unloading complete [ 133.297499] octeon_ep: Unloading ... [ 133.313716] octeon_ep: Unloading complete [ 153.352561] octeon_ep: Loading Marvell Octeon EndPoint NIC Driver ... [ 153.352867] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 153.352871] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 153.352962] octeon_ep: Loaded successfully ! [ 153.362543] octeon_ep_vf: Loading Marvell Octeon EndPoint NIC VF Driver ... [ 158.439781] octeon_ep_vf: Unloading ... [ 158.447417] octeon_ep_vf: Unloading complete [ 158.465311] octeon_ep: Unloading ... [ 158.481453] octeon_ep: Unloading complete [ 178.518149] octeon_ep: Loading Marvell Octeon EndPoint NIC Driver ... [ 178.518522] octeon_ep 0000:87:00.0: Firmware ready status = 0 [ 178.518527] octeon_ep 0000:87:00.0: Firmware not ready; defer probe. [ 178.518626] octeon_ep: Loaded successfully ! [ 178.527976] octeon_ep_vf: Loading Marvell Octeon EndPoint NIC VF Driver ... [ 183.607910] octeon_ep_vf: Unloading ... [ 183.616080] octeon_ep_vf: Unloading complete [ 183.638697] octeon_ep: Unloading ... [ 183.654121] octeon_ep: Unloading complete [ 203.692112] octeon_ep: Loading Marvell Octeon EndPoint NIC Driver ... [ 203.692443] octeon_ep 0000:87:00.0: Firmware ready status = 1 [ 203.692539] octeon_ep 0000:87:00.0: chip_id = 0xb900 [ 203.692541] octeon_ep 0000:87:00.0: Setting up OCTEON CN10KA PF PASS1.0 [ 203.692544] octeon_ep 0000:87:00.0: Octeon device using PCIE Port 0 [ 203.692547] octeon_ep 0000:87:00.0: SDP_EPF_RINFO[0x209f0]:0x8000800000008 [ 203.692550] octeon_ep 0000:87:00.0: SDP_MAC_PF_RING_CTL[0]:0x80001 [ 203.692552] octeon_ep 0000:87:00.0: pf_srn=0 rpvf=8 nvfs=8 rppf=8 [ 203.692579] Octep ctrl mbox : Init successful. [ 203.692580] octeon_ep 0000:87:00.0: Control plane versions host: 10000, firmware: 10000:10000 [ 203.795225] octeon_ep 0000:87:00.0: Heartbeat interval 1000 msecs Heartbeat miss count 20 [ 204.004150] octeon_ep 0000:87:00.0: Device probe successful [ 204.004213] octeon_ep: Loaded successfully ! [ 204.008320] octeon_ep 0000:87:00.0 enp135s0: renamed from eth0 [ 204.014486] octeon_ep_vf: Loading Marvell Octeon EndPoint NIC VF Driver ... [ 204.028913] octeon_ep 0000:87:00.0 enp135s0: Starting netdev ... [ 204.030518] octeon_ep 0000:87:00.0: Allocated 8 IOQ vectors [ 204.031193] octeon_ep 0000:87:00.0: MSI-X enabled successfully [ 204.107141] octeon_ep 0000:87:00.0: Interrupt poll task stopped. [ 209.186665] octeon_ep 0000:87:00.0 enp135s0: NETDEV WATCHDOG: CPU: 34: transmit queue 0 timed out 5149 ms [ 209.187328] octeon_ep 0000:87:00.0 enp135s0: Stopping the device ... [ 209.188063] octeon_ep 0000:87:00.0 enp135s0: IRQs freed [ 209.188462] octeon_ep 0000:87:00.0: Disabled MSI-X [ 209.188473] octeon_ep 0000:87:00.0 enp135s0: Freed IOQ Vectors [ 209.189487] octeon_ep 0000:87:00.0 enp135s0: Device stopped !! [ 209.189490] octeon_ep 0000:87:00.0 enp135s0: Starting netdev ... [ 236.407863] watchdog: BUG: soft lockup - CPU#10 stuck for 26s! [kworker/u160:1:12] [ 236.408266] Modules linked in: octeon_ep_vf octeon_ep nft_reject_inet nf_reject_ipv6 nft_reject veth nft_nat xt_MASQUERADE nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_chain_nat ipt_REJECT nf_reject_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 xt_CT nf_conntrack_netlink xt_REDIRECT xt_nat xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 iTCO_wdt mei_me drm_shmem_helper rapl dell_smbios iTCO_vendor_support intel_cstate dcdbas drm_kms_helper mei dell_wmi_descriptor wmi_bmof pcspkr i2c_i801 intel_uncore cp210x acpi_power_meter lpc_ich i2c_smbus intel_pch_thermal ipmi_si acpi_ipmi ipmi_devintf ipmi_msghandler joydev drm ip_tables overlay loop erofs sr_mod cdrom [ 236.408330] xfs libcrc32c dm_multipath mlx5_ib macsec mlx5_dpll uas usb_storage irdma ice gnss mlx5_core sd_mod sg nvme_tcp nvme_fabrics nvme_keyring ahci libahci nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel i40e nvme_auth libata igb megaraid_sas mlxfw ghash_clmulni_intel psample i2c_algo_bit pci_hyperv_intf wmi libie dca dm_mirror dm_region_hash dm_log dm_mod rpcrdma sunrpc be2iscsi bnx2i cnic uio cxgb4i rdma_ucm ib_uverbs cxgb4 ib_srpt ib_isert iscsi_target_mod tls libcxgbi libcxgb qla4xxx target_core_mod iscsi_boot_sysfs iscsi_tcp libiscsi_tcp ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_core fuse [last unloaded: octeon_ep] [ 236.408382] CPU: 10 PID: 12 Comm: kworker/u160:1 Not tainted 5.14.0-570.32.1.el9_6.x86_64 #1 [ 236.408384] Hardware name: Dell Inc. PowerEdge R740/0YNX56, BIOS 2.21.2 02/19/2024 [ 236.408386] Workqueue: octeon_ep octep_tx_timeout_task [octeon_ep] [ 236.408400] RIP: 0010:octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 236.408410] Code: c6 48 89 fa 48 8b 7c c7 70 48 8b 02 0f b7 48 24 01 f1 48 63 c9 48 c1 e1 11 48 8d b1 00 00 01 00 48 8b 42 38 48 01 f0 48 8b 00 00 00 00 10 74 ef 48 0d 02 00 00 07 48 03 72 38 48 89 06 48 8b [ 236.408412] RSP: 0018:ffffb3c1403cfe00 EFLAGS: 00000286 [ 236.408414] RAX: 0008000007000002 RBX: 0000000000000000 RCX: 0000000000000000 [ 236.408416] RDX: ffff8d8a0af10aa0 RSI: 0000000000010000 RDI: ffffb3c147015000 [ 236.408417] RBP: ffffb3c147015000 R08: ffffb3c14b437000 R09: 000fffffffffffff [ 236.408419] R10: 00000002220bb000 R11: 0000000000000008 R12: 0000000000000000 [ 236.408420] R13: 0000000000010000 R14: 0000000000032000 R15: ffff8d8a0af10aa0 [ 236.408422] FS: 0000000000000000(0000) GS:ffff8d88d0d40000(0000) knlGS:0000000000000000 [ 236.408424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 236.408426] CR2: 000000c0014d7000 CR3: 000000012318e006 CR4: 00000000007726f0 [ 236.408427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 236.408428] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 236.408429] PKRU: 55555554 [ 236.408431] Call Trace: [ 236.408432] [ 236.408434] ? show_trace_log_lvl+0x1c4/0x2df [ 236.408440] ? show_trace_log_lvl+0x1c4/0x2df [ 236.408443] ? octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 236.408452] ? watchdog_timer_fn+0x1ad/0x210 [ 236.408456] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 236.408458] ? __hrtimer_run_queues+0x10f/0x2b0 [ 236.408464] ? hrtimer_interrupt+0xfc/0x210 [ 236.408467] ? __sysvec_apic_timer_interrupt+0x4b/0x100 [ 236.408472] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 236.408477] [ 236.408477] [ 236.408479] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 236.408486] ? octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 236.408494] octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 236.408502] octep_setup_iqs+0x26/0x80 [octeon_ep] [ 236.408511] octep_open+0x44/0x210 [octeon_ep] [ 236.408521] octep_tx_timeout_task.cold+0x10/0x15 [octeon_ep] [ 236.408529] process_one_work+0x194/0x380 [ 236.408533] worker_thread+0x2fe/0x410 [ 236.408535] ? __pfx_worker_thread+0x10/0x10 [ 236.408537] kthread+0xdd/0x100 [ 236.408541] ? __pfx_kthread+0x10/0x10 [ 236.408544] ret_from_fork+0x29/0x50 [ 236.408550] [ 264.405274] watchdog: BUG: soft lockup - CPU#10 stuck for 52s! [kworker/u160:1:12] [ 264.405667] Modules linked in: octeon_ep_vf octeon_ep nft_reject_inet nf_reject_ipv6 nft_reject veth nft_nat xt_MASQUERADE nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_chain_nat ipt_REJECT nf_reject_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 xt_CT nf_conntrack_netlink xt_REDIRECT xt_nat xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 iTCO_wdt mei_me drm_shmem_helper rapl dell_smbios iTCO_vendor_support intel_cstate dcdbas drm_kms_helper mei dell_wmi_descriptor wmi_bmof pcspkr i2c_i801 intel_uncore cp210x acpi_power_meter lpc_ich i2c_smbus intel_pch_thermal ipmi_si acpi_ipmi ipmi_devintf ipmi_msghandler joydev drm ip_tables overlay loop erofs sr_mod cdrom [ 264.405728] xfs libcrc32c dm_multipath mlx5_ib macsec mlx5_dpll uas usb_storage irdma ice gnss mlx5_core sd_mod sg nvme_tcp nvme_fabrics nvme_keyring ahci libahci nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel i40e nvme_auth libata igb megaraid_sas mlxfw ghash_clmulni_intel psample i2c_algo_bit pci_hyperv_intf wmi libie dca dm_mirror dm_region_hash dm_log dm_mod rpcrdma sunrpc be2iscsi bnx2i cnic uio cxgb4i rdma_ucm ib_uverbs cxgb4 ib_srpt ib_isert iscsi_target_mod tls libcxgbi libcxgb qla4xxx target_core_mod iscsi_boot_sysfs iscsi_tcp libiscsi_tcp ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_core fuse [last unloaded: octeon_ep] [ 264.405780] CPU: 10 PID: 12 Comm: kworker/u160:1 Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 264.405783] Hardware name: Dell Inc. PowerEdge R740/0YNX56, BIOS 2.21.2 02/19/2024 [ 264.405785] Workqueue: octeon_ep octep_tx_timeout_task [octeon_ep] [ 264.405799] RIP: 0010:octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 264.405809] Code: c6 48 89 fa 48 8b 7c c7 70 48 8b 02 0f b7 48 24 01 f1 48 63 c9 48 c1 e1 11 48 8d b1 00 00 01 00 48 8b 42 38 48 01 f0 48 8b 00 00 00 00 10 74 ef 48 0d 02 00 00 07 48 03 72 38 48 89 06 48 8b [ 264.405811] RSP: 0018:ffffb3c1403cfe00 EFLAGS: 00000286 [ 264.405813] RAX: 0008000007000002 RBX: 0000000000000000 RCX: 0000000000000000 [ 264.405815] RDX: ffff8d8a0af10aa0 RSI: 0000000000010000 RDI: ffffb3c147015000 [ 264.405816] RBP: ffffb3c147015000 R08: ffffb3c14b437000 R09: 000fffffffffffff [ 264.405818] R10: 00000002220bb000 R11: 0000000000000008 R12: 0000000000000000 [ 264.405819] R13: 0000000000010000 R14: 0000000000032000 R15: ffff8d8a0af10aa0 [ 264.405821] FS: 0000000000000000(0000) GS:ffff8d88d0d40000(0000) knlGS:0000000000000000 [ 264.405822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 264.405824] CR2: 000000c0014d7000 CR3: 000000012318e006 CR4: 00000000007726f0 [ 264.405825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 264.405826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 264.405828] PKRU: 55555554 [ 264.405829] Call Trace: [ 264.405831] [ 264.405833] ? show_trace_log_lvl+0x1c4/0x2df [ 264.405840] ? show_trace_log_lvl+0x1c4/0x2df [ 264.405843] ? octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 264.405851] ? watchdog_timer_fn+0x1ad/0x210 [ 264.405854] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 264.405856] ? __hrtimer_run_queues+0x10f/0x2b0 [ 264.405861] ? hrtimer_interrupt+0xfc/0x210 [ 264.405863] ? __sysvec_apic_timer_interrupt+0x4b/0x100 [ 264.405868] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 264.405873] [ 264.405874] [ 264.405875] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 264.405882] ? octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 264.405890] octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 264.405898] octep_setup_iqs+0x26/0x80 [octeon_ep] [ 264.405906] octep_open+0x44/0x210 [octeon_ep] [ 264.405916] octep_tx_timeout_task.cold+0x10/0x15 [octeon_ep] [ 264.405925] process_one_work+0x194/0x380 [ 264.405928] worker_thread+0x2fe/0x410 [ 264.405931] ? __pfx_worker_thread+0x10/0x10 [ 264.405932] kthread+0xdd/0x100 [ 264.405936] ? __pfx_kthread+0x10/0x10 [ 264.405939] ret_from_fork+0x29/0x50 [ 264.405944] [ 292.402698] watchdog: BUG: soft lockup - CPU#10 stuck for 78s! [kworker/u160:1:12] [ 292.403075] Modules linked in: octeon_ep_vf octeon_ep nft_reject_inet nf_reject_ipv6 nft_reject veth nft_nat xt_MASQUERADE nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_chain_nat ipt_REJECT nf_reject_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 xt_CT nf_conntrack_netlink xt_REDIRECT xt_nat xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 iTCO_wdt mei_me drm_shmem_helper rapl dell_smbios iTCO_vendor_support intel_cstate dcdbas drm_kms_helper mei dell_wmi_descriptor wmi_bmof pcspkr i2c_i801 intel_uncore cp210x acpi_power_meter lpc_ich i2c_smbus intel_pch_thermal ipmi_si acpi_ipmi ipmi_devintf ipmi_msghandler joydev drm ip_tables overlay loop erofs sr_mod cdrom [ 292.403138] xfs libcrc32c dm_multipath mlx5_ib macsec mlx5_dpll uas usb_storage irdma ice gnss mlx5_core sd_mod sg nvme_tcp nvme_fabrics nvme_keyring ahci libahci nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel i40e nvme_auth libata igb megaraid_sas mlxfw ghash_clmulni_intel psample i2c_algo_bit pci_hyperv_intf wmi libie dca dm_mirror dm_region_hash dm_log dm_mod rpcrdma sunrpc be2iscsi bnx2i cnic uio cxgb4i rdma_ucm ib_uverbs cxgb4 ib_srpt ib_isert iscsi_target_mod tls libcxgbi libcxgb qla4xxx target_core_mod iscsi_boot_sysfs iscsi_tcp libiscsi_tcp ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_core fuse [last unloaded: octeon_ep] [ 292.403188] CPU: 10 PID: 12 Comm: kworker/u160:1 Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 292.403191] Hardware name: Dell Inc. PowerEdge R740/0YNX56, BIOS 2.21.2 02/19/2024 [ 292.403193] Workqueue: octeon_ep octep_tx_timeout_task [octeon_ep] [ 292.403207] RIP: 0010:octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 292.403218] Code: c6 48 89 fa 48 8b 7c c7 70 48 8b 02 0f b7 48 24 01 f1 48 63 c9 48 c1 e1 11 48 8d b1 00 00 01 00 48 8b 42 38 48 01 f0 48 8b 00 00 00 00 10 74 ef 48 0d 02 00 00 07 48 03 72 38 48 89 06 48 8b [ 292.403220] RSP: 0018:ffffb3c1403cfe00 EFLAGS: 00000286 [ 292.403222] RAX: 0008000007000002 RBX: 0000000000000000 RCX: 0000000000000000 [ 292.403224] RDX: ffff8d8a0af10aa0 RSI: 0000000000010000 RDI: ffffb3c147015000 [ 292.403225] RBP: ffffb3c147015000 R08: ffffb3c14b437000 R09: 000fffffffffffff [ 292.403227] R10: 00000002220bb000 R11: 0000000000000008 R12: 0000000000000000 [ 292.403228] R13: 0000000000010000 R14: 0000000000032000 R15: ffff8d8a0af10aa0 [ 292.403230] FS: 0000000000000000(0000) GS:ffff8d88d0d40000(0000) knlGS:0000000000000000 [ 292.403231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 292.403233] CR2: 000000c0014d7000 CR3: 000000012318e006 CR4: 00000000007726f0 [ 292.403234] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 292.403235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 292.403237] PKRU: 55555554 [ 292.403238] Call Trace: [ 292.403239] [ 292.403241] ? show_trace_log_lvl+0x1c4/0x2df [ 292.403246] ? show_trace_log_lvl+0x1c4/0x2df [ 292.403249] ? octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 292.403257] ? watchdog_timer_fn+0x1ad/0x210 [ 292.403261] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 292.403262] ? __hrtimer_run_queues+0x10f/0x2b0 [ 292.403268] ? hrtimer_interrupt+0xfc/0x210 [ 292.403270] ? __sysvec_apic_timer_interrupt+0x4b/0x100 [ 292.403274] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 292.403279] [ 292.403280] [ 292.403281] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 292.403288] ? octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 292.403296] octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 292.403304] octep_setup_iqs+0x26/0x80 [octeon_ep] [ 292.403312] octep_open+0x44/0x210 [octeon_ep] [ 292.403322] octep_tx_timeout_task.cold+0x10/0x15 [octeon_ep] [ 292.403330] process_one_work+0x194/0x380 [ 292.403334] worker_thread+0x2fe/0x410 [ 292.403336] ? __pfx_worker_thread+0x10/0x10 [ 292.403338] kthread+0xdd/0x100 [ 292.403342] ? __pfx_kthread+0x10/0x10 [ 292.403344] ret_from_fork+0x29/0x50 [ 292.403350] [ 320.400146] watchdog: BUG: soft lockup - CPU#10 stuck for 104s! [kworker/u160:1:12] [ 320.400517] Modules linked in: octeon_ep_vf octeon_ep nft_reject_inet nf_reject_ipv6 nft_reject veth nft_nat xt_MASQUERADE nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_chain_nat ipt_REJECT nf_reject_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 xt_CT nf_conntrack_netlink xt_REDIRECT xt_nat xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 iTCO_wdt mei_me drm_shmem_helper rapl dell_smbios iTCO_vendor_support intel_cstate dcdbas drm_kms_helper mei dell_wmi_descriptor wmi_bmof pcspkr i2c_i801 intel_uncore cp210x acpi_power_meter lpc_ich i2c_smbus intel_pch_thermal ipmi_si acpi_ipmi ipmi_devintf ipmi_msghandler joydev drm ip_tables overlay loop erofs sr_mod cdrom [ 320.400580] xfs libcrc32c dm_multipath mlx5_ib macsec mlx5_dpll uas usb_storage irdma ice gnss mlx5_core sd_mod sg nvme_tcp nvme_fabrics nvme_keyring ahci libahci nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel i40e nvme_auth libata igb megaraid_sas mlxfw ghash_clmulni_intel psample i2c_algo_bit pci_hyperv_intf wmi libie dca dm_mirror dm_region_hash dm_log dm_mod rpcrdma sunrpc be2iscsi bnx2i cnic uio cxgb4i rdma_ucm ib_uverbs cxgb4 ib_srpt ib_isert iscsi_target_mod tls libcxgbi libcxgb qla4xxx target_core_mod iscsi_boot_sysfs iscsi_tcp libiscsi_tcp ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_core fuse [last unloaded: octeon_ep] [ 320.400631] CPU: 10 PID: 12 Comm: kworker/u160:1 Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 320.400634] Hardware name: Dell Inc. PowerEdge R740/0YNX56, BIOS 2.21.2 02/19/2024 [ 320.400636] Workqueue: octeon_ep octep_tx_timeout_task [octeon_ep] [ 320.400650] RIP: 0010:octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 320.400660] Code: c6 48 89 fa 48 8b 7c c7 70 48 8b 02 0f b7 48 24 01 f1 48 63 c9 48 c1 e1 11 48 8d b1 00 00 01 00 48 8b 42 38 48 01 f0 48 8b 00 00 00 00 10 74 ef 48 0d 02 00 00 07 48 03 72 38 48 89 06 48 8b [ 320.400662] RSP: 0018:ffffb3c1403cfe00 EFLAGS: 00000286 [ 320.400664] RAX: 0008000007000002 RBX: 0000000000000000 RCX: 0000000000000000 [ 320.400666] RDX: ffff8d8a0af10aa0 RSI: 0000000000010000 RDI: ffffb3c147015000 [ 320.400668] RBP: ffffb3c147015000 R08: ffffb3c14b437000 R09: 000fffffffffffff [ 320.400669] R10: 00000002220bb000 R11: 0000000000000008 R12: 0000000000000000 [ 320.400670] R13: 0000000000010000 R14: 0000000000032000 R15: ffff8d8a0af10aa0 [ 320.400672] FS: 0000000000000000(0000) GS:ffff8d88d0d40000(0000) knlGS:0000000000000000 [ 320.400674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 320.400675] CR2: 000000c0014d7000 CR3: 000000012318e006 CR4: 00000000007726f0 [ 320.400677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 320.400678] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 320.400679] PKRU: 55555554 [ 320.400680] Call Trace: [ 320.400682] [ 320.400684] ? show_trace_log_lvl+0x1c4/0x2df [ 320.400690] ? show_trace_log_lvl+0x1c4/0x2df [ 320.400693] ? octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 320.400701] ? watchdog_timer_fn+0x1ad/0x210 [ 320.400704] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 320.400706] ? __hrtimer_run_queues+0x10f/0x2b0 [ 320.400712] ? hrtimer_interrupt+0xfc/0x210 [ 320.400714] ? __sysvec_apic_timer_interrupt+0x4b/0x100 [ 320.400719] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 320.400723] [ 320.400724] [ 320.400725] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 320.400732] ? octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 320.400740] octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 320.400749] octep_setup_iqs+0x26/0x80 [octeon_ep] [ 320.400757] octep_open+0x44/0x210 [octeon_ep] [ 320.400767] octep_tx_timeout_task.cold+0x10/0x15 [octeon_ep] [ 320.400775] process_one_work+0x194/0x380 [ 320.400779] worker_thread+0x2fe/0x410 [ 320.400781] ? __pfx_worker_thread+0x10/0x10 [ 320.400783] kthread+0xdd/0x100 [ 320.400786] ? __pfx_kthread+0x10/0x10 [ 320.400789] ret_from_fork+0x29/0x50 [ 320.400794] [ 348.397594] watchdog: BUG: soft lockup - CPU#10 stuck for 130s! [kworker/u160:1:12] [ 348.397957] Modules linked in: octeon_ep_vf octeon_ep nft_reject_inet nf_reject_ipv6 nft_reject veth nft_nat xt_MASQUERADE nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_chain_nat ipt_REJECT nf_reject_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 xt_CT nf_conntrack_netlink xt_REDIRECT xt_nat xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 iTCO_wdt mei_me drm_shmem_helper rapl dell_smbios iTCO_vendor_support intel_cstate dcdbas drm_kms_helper mei dell_wmi_descriptor wmi_bmof pcspkr i2c_i801 intel_uncore cp210x acpi_power_meter lpc_ich i2c_smbus intel_pch_thermal ipmi_si acpi_ipmi ipmi_devintf ipmi_msghandler joydev drm ip_tables overlay loop erofs sr_mod cdrom [ 348.398019] xfs libcrc32c dm_multipath mlx5_ib macsec mlx5_dpll uas usb_storage irdma ice gnss mlx5_core sd_mod sg nvme_tcp nvme_fabrics nvme_keyring ahci libahci nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel i40e nvme_auth libata igb megaraid_sas mlxfw ghash_clmulni_intel psample i2c_algo_bit pci_hyperv_intf wmi libie dca dm_mirror dm_region_hash dm_log dm_mod rpcrdma sunrpc be2iscsi bnx2i cnic uio cxgb4i rdma_ucm ib_uverbs cxgb4 ib_srpt ib_isert iscsi_target_mod tls libcxgbi libcxgb qla4xxx target_core_mod iscsi_boot_sysfs iscsi_tcp libiscsi_tcp ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_core fuse [last unloaded: octeon_ep] [ 348.398071] CPU: 10 PID: 12 Comm: kworker/u160:1 Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 348.398074] Hardware name: Dell Inc. PowerEdge R740/0YNX56, BIOS 2.21.2 02/19/2024 [ 348.398076] Workqueue: octeon_ep octep_tx_timeout_task [octeon_ep] [ 348.398090] RIP: 0010:octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 348.398100] Code: c6 48 89 fa 48 8b 7c c7 70 48 8b 02 0f b7 48 24 01 f1 48 63 c9 48 c1 e1 11 48 8d b1 00 00 01 00 48 8b 42 38 48 01 f0 48 8b 00 00 00 00 10 74 ef 48 0d 02 00 00 07 48 03 72 38 48 89 06 48 8b [ 348.398102] RSP: 0018:ffffb3c1403cfe00 EFLAGS: 00000286 [ 348.398104] RAX: 0008000007000002 RBX: 0000000000000000 RCX: 0000000000000000 [ 348.398106] RDX: ffff8d8a0af10aa0 RSI: 0000000000010000 RDI: ffffb3c147015000 [ 348.398107] RBP: ffffb3c147015000 R08: ffffb3c14b437000 R09: 000fffffffffffff [ 348.398109] R10: 00000002220bb000 R11: 0000000000000008 R12: 0000000000000000 [ 348.398110] R13: 0000000000010000 R14: 0000000000032000 R15: ffff8d8a0af10aa0 [ 348.398111] FS: 0000000000000000(0000) GS:ffff8d88d0d40000(0000) knlGS:0000000000000000 [ 348.398113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 348.398115] CR2: 000000c0014d7000 CR3: 000000012318e006 CR4: 00000000007726f0 [ 348.398116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 348.398117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 348.398118] PKRU: 55555554 [ 348.398119] Call Trace: [ 348.398121] [ 348.398124] ? show_trace_log_lvl+0x1c4/0x2df [ 348.398129] ? show_trace_log_lvl+0x1c4/0x2df [ 348.398132] ? octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 348.398140] ? watchdog_timer_fn+0x1ad/0x210 [ 348.398143] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 348.398145] ? __hrtimer_run_queues+0x10f/0x2b0 [ 348.398150] ? hrtimer_interrupt+0xfc/0x210 [ 348.398152] ? __sysvec_apic_timer_interrupt+0x4b/0x100 [ 348.398157] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 348.398162] [ 348.398163] [ 348.398164] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 348.398171] ? octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 348.398179] octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 348.398187] octep_setup_iqs+0x26/0x80 [octeon_ep] [ 348.398196] octep_open+0x44/0x210 [octeon_ep] [ 348.398206] octep_tx_timeout_task.cold+0x10/0x15 [octeon_ep] [ 348.398214] process_one_work+0x194/0x380 [ 348.398218] worker_thread+0x2fe/0x410 [ 348.398220] ? __pfx_worker_thread+0x10/0x10 [ 348.398222] kthread+0xdd/0x100 [ 348.398226] ? __pfx_kthread+0x10/0x10 [ 348.398229] ret_from_fork+0x29/0x50 [ 348.398233] [ 369.939973] INFO: task kworker/u161:2:266 blocked for more than 122 seconds. [ 369.940328] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.940656] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.940977] task:kworker/u161:2 state:D stack:0 pid:266 tgid:266 ppid:2 flags:0x00004000 [ 369.940983] Workqueue: events_unbound linkwatch_event [ 369.940991] Call Trace: [ 369.940993] [ 369.940995] __schedule+0x229/0x4a0 [ 369.941002] schedule+0x2e/0xb0 [ 369.941005] schedule_preempt_disabled+0x11/0x20 [ 369.941008] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.941012] ? kfree_rcu_monitor+0xd1/0x270 [ 369.941018] linkwatch_event+0xa/0x30 [ 369.941022] process_one_work+0x194/0x380 [ 369.941025] worker_thread+0x2fe/0x410 [ 369.941028] ? __pfx_worker_thread+0x10/0x10 [ 369.941030] kthread+0xdd/0x100 [ 369.941034] ? __pfx_kthread+0x10/0x10 [ 369.941037] ret_from_fork+0x29/0x50 [ 369.941042] [ 369.941047] INFO: task khugepaged:312 blocked for more than 122 seconds. [ 369.941353] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.941659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.941969] task:khugepaged state:D stack:0 pid:312 tgid:312 ppid:2 flags:0x00004000 [ 369.941973] Call Trace: [ 369.941974] [ 369.941976] __schedule+0x229/0x4a0 [ 369.941981] schedule+0x2e/0xb0 [ 369.941984] schedule_timeout+0x11f/0x160 [ 369.941987] ? sched_clock+0xc/0x30 [ 369.941991] ? sched_clock_cpu+0xb/0x190 [ 369.941996] ? __smp_call_single_queue+0xae/0x120 [ 369.942001] __wait_for_common+0x90/0x1d0 [ 369.942005] ? __pfx_schedule_timeout+0x10/0x10 [ 369.942008] __flush_work+0x16f/0x240 [ 369.942011] ? __pfx_wq_barrier_func+0x10/0x10 [ 369.942016] __lru_add_drain_all+0x178/0x210 [ 369.942022] khugepaged+0x74/0x200 [ 369.942027] ? __pfx_khugepaged+0x10/0x10 [ 369.942029] kthread+0xdd/0x100 [ 369.942032] ? __pfx_kthread+0x10/0x10 [ 369.942034] ret_from_fork+0x29/0x50 [ 369.942038] [ 369.942087] INFO: task ovs-vswitchd:1811 blocked for more than 122 seconds. [ 369.942390] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.942694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.943002] task:ovs-vswitchd state:D stack:0 pid:1811 tgid:1811 ppid:1 flags:0x00404002 [ 369.943006] Call Trace: [ 369.943007] [ 369.943009] __schedule+0x229/0x4a0 [ 369.943013] schedule+0x2e/0xb0 [ 369.943016] schedule_preempt_disabled+0x11/0x20 [ 369.943019] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.943024] rtnetlink_rcv_msg+0x111/0x410 [ 369.943027] ? __wake_up+0x40/0x60 [ 369.943031] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 369.943034] netlink_rcv_skb+0x54/0x100 [ 369.943039] netlink_unicast+0x245/0x370 [ 369.943042] netlink_sendmsg+0x206/0x440 [ 369.943045] ____sys_sendmsg+0x388/0x3b0 [ 369.943049] ? import_iovec+0x17/0x20 [ 369.943053] ? copy_msghdr_from_user+0x6d/0xa0 [ 369.943058] ___sys_sendmsg+0x88/0xd0 [ 369.943062] ? exit_to_user_mode_prepare+0xde/0x100 [ 369.943065] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.943070] ? do_syscall_64+0x6b/0xe0 [ 369.943073] ? exit_to_user_mode_prepare+0xde/0x100 [ 369.943075] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.943078] ? do_syscall_64+0x6b/0xe0 [ 369.943079] ? __fget_light+0x9f/0x130 [ 369.943085] __sys_sendmsg+0x59/0xa0 [ 369.943089] do_syscall_64+0x5c/0xe0 [ 369.943092] ? __rseq_handle_notify_resume+0x26/0xc0 [ 369.943096] ? exit_to_user_mode_loop+0xd9/0x130 [ 369.943098] ? exit_to_user_mode_prepare+0xde/0x100 [ 369.943100] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.943103] ? do_syscall_64+0x6b/0xe0 [ 369.943104] ? do_syscall_64+0x6b/0xe0 [ 369.943106] ? clear_bhb_loop+0x25/0x80 [ 369.943110] ? clear_bhb_loop+0x25/0x80 [ 369.943113] ? clear_bhb_loop+0x25/0x80 [ 369.943115] ? clear_bhb_loop+0x25/0x80 [ 369.943117] ? clear_bhb_loop+0x25/0x80 [ 369.943120] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.943123] RIP: 0033:0x7f7dc74e8f1d [ 369.943125] RSP: 002b:00007ffd208dc870 EFLAGS: 00000293 ORIG_RAX: 000000000000002e [ 369.943128] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7dc74e8f1d [ 369.943130] RDX: 0000000000000000 RSI: 00007ffd208dc900 RDI: 0000000000000012 [ 369.943132] RBP: 00007ffd208dd638 R08: 0000000000000000 R09: 0000000000000020 [ 369.943133] R10: 0000000000000000 R11: 0000000000000293 R12: 000055ef450bea40 [ 369.943134] R13: 0000000000000001 R14: 00000000000006d4 R15: 00007ffd208dc900 [ 369.943139] [ 369.943147] INFO: task NetworkManager:1887 blocked for more than 122 seconds. [ 369.943450] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.943754] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.944067] task:NetworkManager state:D stack:0 pid:1887 tgid:1887 ppid:1 flags:0x00000002 [ 369.944071] Call Trace: [ 369.944072] [ 369.944073] __schedule+0x229/0x4a0 [ 369.944078] schedule+0x2e/0xb0 [ 369.944081] schedule_preempt_disabled+0x11/0x20 [ 369.944084] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.944088] ? rcutree_enqueue+0x23/0x140 [ 369.944091] ip6mr_sk_done+0x2e/0xe0 [ 369.944097] rawv6_close+0x19/0x30 [ 369.944100] inet_release+0x3c/0x80 [ 369.944105] __sock_release+0x3a/0xb0 [ 369.944110] sock_close+0x14/0x20 [ 369.944113] __fput+0x91/0x250 [ 369.944118] __x64_sys_close+0x39/0x80 [ 369.944122] do_syscall_64+0x5c/0xe0 [ 369.944125] ? kmem_cache_free+0x37a/0x420 [ 369.944128] ? __fput+0xff/0x250 [ 369.944131] ? kmem_cache_free+0x37a/0x420 [ 369.944133] ? _copy_from_user+0x33/0x70 [ 369.944137] ? eventfd_write+0xc7/0x210 [ 369.944142] ? vfs_write+0xeb/0x480 [ 369.944145] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.944148] ? do_syscall_64+0x6b/0xe0 [ 369.944150] ? syscall_exit_work+0x103/0x130 [ 369.944152] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.944156] ? __fget_light+0x9f/0x130 [ 369.944159] ? syscall_exit_work+0x103/0x130 [ 369.944161] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.944164] ? clear_bhb_loop+0x25/0x80 [ 369.944168] ? clear_bhb_loop+0x25/0x80 [ 369.944170] ? clear_bhb_loop+0x25/0x80 [ 369.944173] ? clear_bhb_loop+0x25/0x80 [ 369.944175] ? clear_bhb_loop+0x25/0x80 [ 369.944178] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.944181] RIP: 0033:0x7f731ec2273b [ 369.944182] RSP: 002b:00007ffdec0a2720 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 [ 369.944185] RAX: ffffffffffffffda RBX: 000055f54919cb80 RCX: 00007f731ec2273b [ 369.944186] RDX: 0000000000000002 RSI: 0000000000000002 RDI: 000000000000001a [ 369.944188] RBP: 000055f5492f12d0 R08: 0000000000000000 R09: 0000000000000016 [ 369.944189] R10: 0000000000000000 R11: 0000000000000293 R12: 00007ffdec0a2788 [ 369.944190] R13: 0000000000000000 R14: 000055f5492768b0 R15: 000055f54919cb80 [ 369.944193] [ 369.944229] INFO: task kubelet:4315 blocked for more than 122 seconds. [ 369.944534] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.944844] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.945166] task:kubelet state:D stack:0 pid:4315 tgid:4249 ppid:1 flags:0x00000002 [ 369.945170] Call Trace: [ 369.945171] [ 369.945172] __schedule+0x229/0x4a0 [ 369.945176] schedule+0x2e/0xb0 [ 369.945179] schedule_preempt_disabled+0x11/0x20 [ 369.945183] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.945187] ? __pfx_rtnl_dump_all+0x10/0x10 [ 369.945189] rtnl_dumpit+0x70/0xa0 [ 369.945194] netlink_dump+0x1a3/0x320 [ 369.945201] __netlink_dump_start+0x1e1/0x300 [ 369.945204] ? __pfx_rtnl_dump_all+0x10/0x10 [ 369.945206] rtnetlink_rcv_msg+0x2ba/0x410 [ 369.945209] ? __pfx_rtnl_dumpit+0x10/0x10 [ 369.945212] ? __pfx_rtnl_dump_all+0x10/0x10 [ 369.945214] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 369.945216] netlink_rcv_skb+0x54/0x100 [ 369.945220] netlink_unicast+0x245/0x370 [ 369.945223] netlink_sendmsg+0x206/0x440 [ 369.945225] __sys_sendto+0x1dc/0x1f0 [ 369.945230] ? release_sock+0x19/0x90 [ 369.945235] __x64_sys_sendto+0x20/0x30 [ 369.945237] do_syscall_64+0x5c/0xe0 [ 369.945240] ? __sys_bind+0xe1/0x100 [ 369.945243] ? file_init_path+0x6b/0xf0 [ 369.945246] ? alloc_file_pseudo+0xac/0x100 [ 369.945249] ? syscall_exit_work+0x103/0x130 [ 369.945251] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.945254] ? do_syscall_64+0x6b/0xe0 [ 369.945256] ? __sys_socket+0xcd/0x110 [ 369.945258] ? syscall_exit_work+0x103/0x130 [ 369.945260] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.945263] ? do_syscall_64+0x6b/0xe0 [ 369.945265] ? futex_wake+0x155/0x190 [ 369.945269] ? do_futex+0xbe/0x1d0 [ 369.945273] ? __x64_sys_futex+0x73/0x1d0 [ 369.945276] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.945279] ? do_syscall_64+0x6b/0xe0 [ 369.945281] ? syscall_exit_work+0x103/0x130 [ 369.945283] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.945286] ? do_syscall_64+0x6b/0xe0 [ 369.945288] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.945291] ? do_syscall_64+0x6b/0xe0 [ 369.945293] ? exit_to_user_mode_prepare+0xef/0x100 [ 369.945295] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.945298] ? do_syscall_64+0x6b/0xe0 [ 369.945300] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.945303] ? do_syscall_64+0x6b/0xe0 [ 369.945305] ? do_syscall_64+0x6b/0xe0 [ 369.945306] ? clear_bhb_loop+0x25/0x80 [ 369.945309] ? clear_bhb_loop+0x25/0x80 [ 369.945312] ? clear_bhb_loop+0x25/0x80 [ 369.945314] ? clear_bhb_loop+0x25/0x80 [ 369.945317] ? clear_bhb_loop+0x25/0x80 [ 369.945319] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.945322] RIP: 0033:0x484dae [ 369.945323] RSP: 002b:000000c003183550 EFLAGS: 00000212 ORIG_RAX: 000000000000002c [ 369.945326] RAX: ffffffffffffffda RBX: 0000000000000021 RCX: 0000000000484dae [ 369.945327] RDX: 0000000000000011 RSI: 000000c001fed350 RDI: 0000000000000021 [ 369.945328] RBP: 000000c003183590 R08: 000000c001fed344 R09: 000000000000000c [ 369.945329] R10: 0000000000000000 R11: 0000000000000212 R12: 000000c001fed350 [ 369.945330] R13: 0000000000000000 R14: 000000c00101afc0 R15: 00017f7efdffffff [ 369.945333] [ 369.945353] INFO: task dynkeepalived:7344 blocked for more than 122 seconds. [ 369.945676] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.946002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.946320] task:dynkeepalived state:D stack:0 pid:7344 tgid:4553 ppid:4551 flags:0x00000002 [ 369.946323] Call Trace: [ 369.946324] [ 369.946326] __schedule+0x229/0x4a0 [ 369.946330] schedule+0x2e/0xb0 [ 369.946333] schedule_preempt_disabled+0x11/0x20 [ 369.946336] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.946340] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.946343] rtnl_dumpit+0x70/0xa0 [ 369.946347] netlink_dump+0x1a3/0x320 [ 369.946353] __netlink_dump_start+0x1e1/0x300 [ 369.946356] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.946359] rtnetlink_rcv_msg+0x2ba/0x410 [ 369.946361] ? __pfx_rtnl_dumpit+0x10/0x10 [ 369.946364] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.946367] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 369.946369] netlink_rcv_skb+0x54/0x100 [ 369.946373] netlink_unicast+0x245/0x370 [ 369.946375] netlink_sendmsg+0x206/0x440 [ 369.946378] __sys_sendto+0x1dc/0x1f0 [ 369.946382] ? syscall_exit_work+0x103/0x130 [ 369.946384] __x64_sys_sendto+0x20/0x30 [ 369.946387] do_syscall_64+0x5c/0xe0 [ 369.946389] ? generic_exec_single+0x35/0x130 [ 369.946393] ? smp_call_function_single_async+0x1e/0x40 [ 369.946395] ? update_process_times+0x7f/0x90 [ 369.946399] ? tick_sched_handle+0x21/0x60 [ 369.946401] ? __pfx_tick_nohz_highres_handler+0x10/0x10 [ 369.946403] ? timerqueue_add+0x9b/0xc0 [ 369.946408] ? enqueue_hrtimer+0x35/0x90 [ 369.946411] ? sched_clock+0xc/0x30 [ 369.946414] ? sched_clock_cpu+0xb/0x190 [ 369.946418] ? irqtime_account_irq+0x3c/0xb0 [ 369.946423] ? handle_softirqs+0x16b/0x270 [ 369.946429] ? clear_bhb_loop+0x25/0x80 [ 369.946432] ? clear_bhb_loop+0x25/0x80 [ 369.946434] ? clear_bhb_loop+0x25/0x80 [ 369.946437] ? clear_bhb_loop+0x25/0x80 [ 369.946439] ? clear_bhb_loop+0x25/0x80 [ 369.946442] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.946445] RIP: 0033:0x47ffce [ 369.946446] RSP: 002b:000000c00053a180 EFLAGS: 00000216 ORIG_RAX: 000000000000002c [ 369.946448] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 000000000047ffce [ 369.946449] RDX: 0000000000000011 RSI: 000000c000236d80 RDI: 0000000000000007 [ 369.946450] RBP: 000000c00053a1c0 R08: 000000c000236d74 R09: 000000000000000c [ 369.946451] R10: 0000000000000000 R11: 0000000000000216 R12: 000000c000236d80 [ 369.946453] R13: 0000000000000000 R14: 000000c0000081c0 R15: 000073fffffffffd [ 369.946455] [ 369.946461] INFO: task corednsmonitor:4578 blocked for more than 122 seconds. [ 369.946780] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.947105] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.947432] task:corednsmonitor state:D stack:0 pid:4578 tgid:4560 ppid:4558 flags:0x00000002 [ 369.947436] Call Trace: [ 369.947436] [ 369.947438] __schedule+0x229/0x4a0 [ 369.947442] schedule+0x2e/0xb0 [ 369.947445] schedule_preempt_disabled+0x11/0x20 [ 369.947447] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.947451] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.947455] rtnl_dumpit+0x70/0xa0 [ 369.947459] netlink_dump+0x1a3/0x320 [ 369.947464] __netlink_dump_start+0x1e1/0x300 [ 369.947466] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.947469] rtnetlink_rcv_msg+0x2ba/0x410 [ 369.947472] ? __pfx_rtnl_dumpit+0x10/0x10 [ 369.947475] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.947478] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 369.947480] netlink_rcv_skb+0x54/0x100 [ 369.947484] netlink_unicast+0x245/0x370 [ 369.947486] netlink_sendmsg+0x206/0x440 [ 369.947489] __sys_sendto+0x1dc/0x1f0 [ 369.947493] ? __wake_up+0x40/0x60 [ 369.947496] ? netlink_bind+0x2ff/0x360 [ 369.947498] __x64_sys_sendto+0x20/0x30 [ 369.947501] do_syscall_64+0x5c/0xe0 [ 369.947504] ? syscall_exit_work+0x103/0x130 [ 369.947505] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.947509] ? do_syscall_64+0x6b/0xe0 [ 369.947510] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.947514] ? do_syscall_64+0x6b/0xe0 [ 369.947515] ? syscall_exit_work+0x103/0x130 [ 369.947517] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.947520] ? clear_bhb_loop+0x25/0x80 [ 369.947523] ? clear_bhb_loop+0x25/0x80 [ 369.947525] ? clear_bhb_loop+0x25/0x80 [ 369.947527] ? clear_bhb_loop+0x25/0x80 [ 369.947530] ? clear_bhb_loop+0x25/0x80 [ 369.947532] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.947535] RIP: 0033:0x47fe0e [ 369.947536] RSP: 002b:000000c000621580 EFLAGS: 00000216 ORIG_RAX: 000000000000002c [ 369.947538] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 000000000047fe0e [ 369.947539] RDX: 0000000000000011 RSI: 000000c000142198 RDI: 0000000000000006 [ 369.947540] RBP: 000000c0006215c0 R08: 000000c00014218c R09: 000000000000000c [ 369.947542] R10: 0000000000000000 R11: 0000000000000216 R12: 000000c000142198 [ 369.947543] R13: 0000000000000000 R14: 000000c000008380 R15: 0000000020001e94 [ 369.947546] [ 369.947577] INFO: task node_exporter:5106 blocked for more than 122 seconds. [ 369.947905] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.948240] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.948574] task:node_exporter state:D stack:0 pid:5106 tgid:5106 ppid:5102 flags:0x00004002 [ 369.948577] Call Trace: [ 369.948578] [ 369.948580] __schedule+0x229/0x4a0 [ 369.948584] schedule+0x2e/0xb0 [ 369.948587] schedule_preempt_disabled+0x11/0x20 [ 369.948589] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.948593] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.948597] rtnl_dumpit+0x70/0xa0 [ 369.948600] netlink_dump+0x1a3/0x320 [ 369.948605] __netlink_dump_start+0x1e1/0x300 [ 369.948608] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.948610] rtnetlink_rcv_msg+0x2ba/0x410 [ 369.948613] ? __pfx_rtnl_dumpit+0x10/0x10 [ 369.948616] ? __pfx_rtnl_dump_ifinfo+0x10/0x10 [ 369.948619] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 369.948621] netlink_rcv_skb+0x54/0x100 [ 369.948625] netlink_unicast+0x245/0x370 [ 369.948627] netlink_sendmsg+0x206/0x440 [ 369.948630] ____sys_sendmsg+0x388/0x3b0 [ 369.948632] ? import_iovec+0x17/0x20 [ 369.948635] ? copy_msghdr_from_user+0x6d/0xa0 [ 369.948639] ___sys_sendmsg+0x88/0xd0 [ 369.948642] ? avc_has_perm_noaudit+0x94/0x110 [ 369.948647] ? __wake_up+0x40/0x60 [ 369.948650] ? netlink_bind+0x2ff/0x360 [ 369.948652] ? avc_has_perm+0x48/0xd0 [ 369.948655] ? __fget_light+0x9f/0x130 [ 369.948659] __sys_sendmsg+0x59/0xa0 [ 369.948662] do_syscall_64+0x5c/0xe0 [ 369.948665] ? syscall_exit_work+0x103/0x130 [ 369.948667] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.948670] ? do_syscall_64+0x6b/0xe0 [ 369.948672] ? file_has_perm+0x8c/0xd0 [ 369.948677] ? syscall_exit_work+0x103/0x130 [ 369.948678] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.948681] ? do_syscall_64+0x6b/0xe0 [ 369.948683] ? exit_to_user_mode_prepare+0xb9/0x100 [ 369.948685] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.948688] ? do_syscall_64+0x6b/0xe0 [ 369.948690] ? do_syscall_64+0x6b/0xe0 [ 369.948692] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.948694] ? clear_bhb_loop+0x25/0x80 [ 369.948698] ? clear_bhb_loop+0x25/0x80 [ 369.948700] ? clear_bhb_loop+0x25/0x80 [ 369.948702] ? clear_bhb_loop+0x25/0x80 [ 369.948705] ? clear_bhb_loop+0x25/0x80 [ 369.948707] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.948710] RIP: 0033:0x483fce [ 369.948711] RSP: 002b:000000c00050d100 EFLAGS: 00000212 ORIG_RAX: 000000000000002e [ 369.948713] RAX: ffffffffffffffda RBX: 000000000000000e RCX: 0000000000483fce [ 369.948714] RDX: 0000000000000000 RSI: 000000c00050d260 RDI: 000000000000000e [ 369.948715] RBP: 000000c00050d140 R08: 0000000000000000 R09: 0000000000000000 [ 369.948717] R10: 0000000000000000 R11: 0000000000000212 R12: 0000000000000000 [ 369.948718] R13: 000000c000057d38 R14: 000000c000168700 R15: 0007ffffffffffff [ 369.948720] [ 369.948722] INFO: task node_exporter:5207 blocked for more than 122 seconds. [ 369.949066] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.949411] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.949759] task:node_exporter state:D stack:0 pid:5207 tgid:5106 ppid:5102 flags:0x00000002 [ 369.949762] Call Trace: [ 369.949763] [ 369.949764] __schedule+0x229/0x4a0 [ 369.949768] schedule+0x2e/0xb0 [ 369.949771] schedule_preempt_disabled+0x11/0x20 [ 369.949774] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.949778] ethnl_default_dumpit+0xb4/0x230 [ 369.949783] genl_dumpit+0x31/0x90 [ 369.949786] netlink_dump+0x1a3/0x320 [ 369.949791] __netlink_dump_start+0x1e1/0x300 [ 369.949793] genl_family_rcv_msg_dumpit+0x93/0xf0 [ 369.949796] ? __pfx_ethnl_default_start+0x10/0x10 [ 369.949799] ? __pfx_ethnl_default_dumpit+0x10/0x10 [ 369.949803] ? __pfx_genl_start+0x10/0x10 [ 369.949805] ? __pfx_genl_dumpit+0x10/0x10 [ 369.949807] ? __pfx_genl_done+0x10/0x10 [ 369.949810] genl_family_rcv_msg+0xf1/0x220 [ 369.949812] ? __pfx_ethnl_default_start+0x10/0x10 [ 369.949815] ? __pfx_ethnl_default_dumpit+0x10/0x10 [ 369.949818] ? __pfx_ethnl_default_done+0x10/0x10 [ 369.949821] ? pskb_expand_head+0x8b/0x360 [ 369.949825] ? ep_poll_callback+0x25d/0x2a0 [ 369.949830] genl_rcv_msg+0x47/0xa0 [ 369.949833] ? __pfx_genl_rcv_msg+0x10/0x10 [ 369.949835] netlink_rcv_skb+0x54/0x100 [ 369.949838] genl_rcv+0x24/0x40 [ 369.949840] netlink_unicast+0x245/0x370 [ 369.949843] netlink_sendmsg+0x206/0x440 [ 369.949845] ____sys_sendmsg+0x388/0x3b0 [ 369.949848] ? import_iovec+0x17/0x20 [ 369.949850] ? copy_msghdr_from_user+0x6d/0xa0 [ 369.949853] ___sys_sendmsg+0x88/0xd0 [ 369.949856] ? __wake_up+0x40/0x60 [ 369.949859] ? netlink_recvmsg+0x214/0x290 [ 369.949863] ? __check_object_size.part.0+0x35/0xd0 [ 369.949868] ? _copy_to_user+0x2a/0x40 [ 369.949871] ? move_addr_to_user+0x4b/0xe0 [ 369.949874] ? ____sys_recvmsg+0xeb/0x1b0 [ 369.949876] ? __fget_light+0x9f/0x130 [ 369.949880] __sys_sendmsg+0x59/0xa0 [ 369.949884] do_syscall_64+0x5c/0xe0 [ 369.949887] ? import_iovec+0x17/0x20 [ 369.949889] ? copy_msghdr_from_user+0x6d/0xa0 [ 369.949892] ? __fget_light+0x9f/0x130 [ 369.949895] ? __sys_recvmsg+0x89/0xa0 [ 369.949899] ? syscall_exit_work+0x103/0x130 [ 369.949901] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.949904] ? do_syscall_64+0x6b/0xe0 [ 369.949906] ? __sys_recvmsg+0x89/0xa0 [ 369.949910] ? syscall_exit_work+0x103/0x130 [ 369.949911] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.949914] ? do_syscall_64+0x6b/0xe0 [ 369.949916] ? syscall_exit_work+0x103/0x130 [ 369.949918] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.949921] ? do_syscall_64+0x6b/0xe0 [ 369.949923] ? __sys_setsockopt+0x75/0xc0 [ 369.949926] ? __sys_setsockopt+0xad/0xc0 [ 369.949929] ? syscall_exit_work+0x103/0x130 [ 369.949935] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.949938] ? do_syscall_64+0x6b/0xe0 [ 369.949940] ? syscall_exit_work+0x103/0x130 [ 369.949942] ? clear_bhb_loop+0x25/0x80 [ 369.949945] ? clear_bhb_loop+0x25/0x80 [ 369.949947] ? clear_bhb_loop+0x25/0x80 [ 369.949950] ? clear_bhb_loop+0x25/0x80 [ 369.949953] ? clear_bhb_loop+0x25/0x80 [ 369.949955] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.949958] RIP: 0033:0x483fce [ 369.949959] RSP: 002b:000000c000178d40 EFLAGS: 00000216 ORIG_RAX: 000000000000002e [ 369.949961] RAX: ffffffffffffffda RBX: 000000000000000c RCX: 0000000000483fce [ 369.949963] RDX: 0000000000000000 RSI: 000000c000178ea0 RDI: 000000000000000c [ 369.949964] RBP: 000000c000178d80 R08: 0000000000000000 R09: 0000000000000000 [ 369.949966] R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000 [ 369.949967] R13: 000000c00005ab50 R14: 000000c0000dc700 R15: 00001fffffffffe1 [ 369.949969] [ 369.949970] INFO: task node_exporter:5208 blocked for more than 122 seconds. [ 369.950324] Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 369.950681] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.951041] task:node_exporter state:D stack:0 pid:5208 tgid:5106 ppid:5102 flags:0x00000002 [ 369.951044] Call Trace: [ 369.951046] [ 369.951047] __schedule+0x229/0x4a0 [ 369.951051] schedule+0x2e/0xb0 [ 369.951054] schedule_preempt_disabled+0x11/0x20 [ 369.951057] __mutex_lock.constprop.0+0x433/0x6a0 [ 369.951062] ? __pfx_neigh_dump_info+0x10/0x10 [ 369.951065] rtnl_dumpit+0x70/0xa0 [ 369.951069] netlink_dump+0x1a3/0x320 [ 369.951074] __netlink_dump_start+0x1e1/0x300 [ 369.951077] ? __pfx_neigh_dump_info+0x10/0x10 [ 369.951080] rtnetlink_rcv_msg+0x2ba/0x410 [ 369.951083] ? __pfx_rtnl_dumpit+0x10/0x10 [ 369.951086] ? __pfx_neigh_dump_info+0x10/0x10 [ 369.951089] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 369.951091] netlink_rcv_skb+0x54/0x100 [ 369.951095] netlink_unicast+0x245/0x370 [ 369.951097] netlink_sendmsg+0x206/0x440 [ 369.951100] ____sys_sendmsg+0x388/0x3b0 [ 369.951102] ? import_iovec+0x17/0x20 [ 369.951104] ? copy_msghdr_from_user+0x6d/0xa0 [ 369.951108] ___sys_sendmsg+0x88/0xd0 [ 369.951111] ? __wake_up+0x40/0x60 [ 369.951114] ? netlink_bind+0x2ff/0x360 [ 369.951116] ? __sys_bind+0xe1/0x100 [ 369.951119] ? __fget_light+0x9f/0x130 [ 369.951122] __sys_sendmsg+0x59/0xa0 [ 369.951126] do_syscall_64+0x5c/0xe0 [ 369.951128] ? syscall_exit_work+0x103/0x130 [ 369.951130] ? syscall_exit_to_user_mode+0x19/0x40 [ 369.951133] ? clear_bhb_loop+0x25/0x80 [ 369.951136] ? clear_bhb_loop+0x25/0x80 [ 369.951139] ? clear_bhb_loop+0x25/0x80 [ 369.951141] ? clear_bhb_loop+0x25/0x80 [ 369.951144] ? clear_bhb_loop+0x25/0x80 [ 369.951146] entry_SYSCALL_64_after_hwframe+0x78/0x80 [ 369.951149] RIP: 0033:0x483fce [ 369.951150] RSP: 002b:000000c00017d040 EFLAGS: 00000216 ORIG_RAX: 000000000000002e [ 369.951152] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 0000000000483fce [ 369.951153] RDX: 0000000000000000 RSI: 000000c00017d1a0 RDI: 000000000000000a [ 369.951154] RBP: 000000c00017d080 R08: 0000000000000000 R09: 0000000000000000 [ 369.951155] R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000 [ 369.951157] R13: 0000000000000000 R14: 000000c00031ac40 R15: 1fffffffffffffff [ 369.951159] [ 376.395076] watchdog: BUG: soft lockup - CPU#10 stuck for 156s! [kworker/u160:1:12] [ 376.395503] Modules linked in: octeon_ep_vf octeon_ep nft_reject_inet nf_reject_ipv6 nft_reject veth nft_nat xt_MASQUERADE nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_chain_nat ipt_REJECT nf_reject_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 xt_CT nf_conntrack_netlink xt_REDIRECT xt_nat xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 iTCO_wdt mei_me drm_shmem_helper rapl dell_smbios iTCO_vendor_support intel_cstate dcdbas drm_kms_helper mei dell_wmi_descriptor wmi_bmof pcspkr i2c_i801 intel_uncore cp210x acpi_power_meter lpc_ich i2c_smbus intel_pch_thermal ipmi_si acpi_ipmi ipmi_devintf ipmi_msghandler joydev drm ip_tables overlay loop erofs sr_mod cdrom [ 376.395566] xfs libcrc32c dm_multipath mlx5_ib macsec mlx5_dpll uas usb_storage irdma ice gnss mlx5_core sd_mod sg nvme_tcp nvme_fabrics nvme_keyring ahci libahci nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel i40e nvme_auth libata igb megaraid_sas mlxfw ghash_clmulni_intel psample i2c_algo_bit pci_hyperv_intf wmi libie dca dm_mirror dm_region_hash dm_log dm_mod rpcrdma sunrpc be2iscsi bnx2i cnic uio cxgb4i rdma_ucm ib_uverbs cxgb4 ib_srpt ib_isert iscsi_target_mod tls libcxgbi libcxgb qla4xxx target_core_mod iscsi_boot_sysfs iscsi_tcp libiscsi_tcp ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_core fuse [last unloaded: octeon_ep] [ 376.395616] CPU: 10 PID: 12 Comm: kworker/u160:1 Tainted: G L ------- --- 5.14.0-570.32.1.el9_6.x86_64 #1 [ 376.395619] Hardware name: Dell Inc. PowerEdge R740/0YNX56, BIOS 2.21.2 02/19/2024 [ 376.395621] Workqueue: octeon_ep octep_tx_timeout_task [octeon_ep] [ 376.395634] RIP: 0010:octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 376.395645] Code: c6 48 89 fa 48 8b 7c c7 70 48 8b 02 0f b7 48 24 01 f1 48 63 c9 48 c1 e1 11 48 8d b1 00 00 01 00 48 8b 42 38 48 01 f0 48 8b 00 00 00 00 10 74 ef 48 0d 02 00 00 07 48 03 72 38 48 89 06 48 8b [ 376.395647] RSP: 0018:ffffb3c1403cfe00 EFLAGS: 00000286 [ 376.395649] RAX: 0008000007000002 RBX: 0000000000000000 RCX: 0000000000000000 [ 376.395651] RDX: ffff8d8a0af10aa0 RSI: 0000000000010000 RDI: ffffb3c147015000 [ 376.395652] RBP: ffffb3c147015000 R08: ffffb3c14b437000 R09: 000fffffffffffff [ 376.395653] R10: 00000002220bb000 R11: 0000000000000008 R12: 0000000000000000 [ 376.395655] R13: 0000000000010000 R14: 0000000000032000 R15: ffff8d8a0af10aa0 [ 376.395656] FS: 0000000000000000(0000) GS:ffff8d88d0d40000(0000) knlGS:0000000000000000 [ 376.395658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 376.395660] CR2: 000000c0014d7000 CR3: 000000012318e006 CR4: 00000000007726f0 [ 376.395661] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 376.395662] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 376.395663] PKRU: 55555554 [ 376.395665] Call Trace: [ 376.395667] [ 376.395669] ? show_trace_log_lvl+0x1c4/0x2df [ 376.395675] ? show_trace_log_lvl+0x1c4/0x2df [ 376.395678] ? octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 376.395686] ? watchdog_timer_fn+0x1ad/0x210 [ 376.395689] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 376.395691] ? __hrtimer_run_queues+0x10f/0x2b0 [ 376.395696] ? hrtimer_interrupt+0xfc/0x210 [ 376.395699] ? __sysvec_apic_timer_interrupt+0x4b/0x100 [ 376.395703] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 376.395708] [ 376.395709] [ 376.395710] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 376.395717] ? octep_setup_iq_regs_cnxk_pf+0x31/0xc0 [octeon_ep] [ 376.395725] octep_setup_iq+0x182/0x1b0 [octeon_ep] [ 376.395733] octep_setup_iqs+0x26/0x80 [octeon_ep] [ 376.395741] octep_open+0x44/0x210 [octeon_ep] [ 376.395751] octep_tx_timeout_task.cold+0x10/0x15 [octeon_ep] [ 376.395760] process_one_work+0x194/0x380 [ 376.395763] worker_thread+0x2fe/0x410 [ 376.395765] ? __pfx_worker_thread+0x10/0x10 [ 376.395767] kthread+0xdd/0x100 [ 376.395771] ? __pfx_kthread+0x10/0x10 [ 376.395774] ret_from_fork+0x29/0x50 [ 376.395779]