Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-17895

[tuned] Failure in moving IRQ threads to housekeeping CPUs from isolated CPUs

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • rhel-8.10.z
    • tuned
    • None
    • Important
    • rhel-sst-cs-net-perf-services
    • ssg_core_services
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • x86_64
    • None

      What were you trying to do that didn't work?

      Start cpu-partitioning profile failed.

      Please provide the package NVR for which bug is seen:

      RHEL-9.4.0-updates-20231128.49

      5.14.0-390.el9.x86_64

      tuned-2.21.0-1.el9_3.noarch

      tuned-profiles-cpu-partitioning-2.21.0-1.el9_3.noarch

       

      test machine: dell-per750-65.rhts.eng.pek2.redhat.com

      How reproducible: 100%

      Steps to reproduce

      1. yum reinstall tuned-profiles-cpu-partitioning
      2. tuned-adm profile cpu-partitioning
        CONSOLE  tuned.plugins.plugin_systemd: you may need to manualy run 'dracut -f' to update the systemd configuration in initrd image
      3. dracut -f
      4. reboot
      5. cat /proc/cmdline 
        BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-390.el9.x86_64 root=/dev/mapper/rhel_dell-per75065-root ro ksdevice=bootif pci=realloc crashkernel=1G-4G:192M,4G-64G:256M,64G:512M resume=/dev/mapper/rhel_dell-per750-65-swap rd.lvm.lv=rhel_dell-per750-65/root rd.lvm.lv=rhel_dell-per750-65/swap console=ttyS0,115200n81
      6. cat /etc/tuned/cpu-partitioning-variables.conf 
        Examples:
        isolated_cores=2,4-7
        isolated_cores=2-23

      Reserve 1 core per socket for housekeeping, isolate the rest.
      isolated_cores=${f:calc_isolated_cores:1}
      To disable the kernel load balancing in certain isolated CPUs:
      no_balance_cores=5-10

      check log
      2023-12-01 11:07:09,407 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 1180 to '00000002': [Errno 28] No space left on device
      2023-12-01 11:07:09,407 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 1181 to '00000002': [Errno 28] No space left on device
      2023-12-01 11:07:09,452 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/cpu-partitioning/script.sh' with arguments '['start']'
      2023-12-01 11:07:09,649 INFO     tuned.plugins.plugin_bootloader: generating initrd image from directory '/tmp/tmp.aVsMAlkc0K'
      2023-12-01 11:07:09,679 INFO     tuned.plugins.plugin_bootloader: installing initrd image as '/boot/tuned-initrd.img'
      2023-12-01 11:07:09,695 INFO     tuned.plugins.plugin_bootloader: removing directory '/tmp/tmp.aVsMAlkc0K'
      2023-12-01 11:07:09,697 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
      2023-12-01 11:07:09,743 ERROR    tuned.utils.commands: Executing grub2-editenv error: grub2-editenv: error: environment block too small.
      2023-12-01 11:07:09,743 WARNING  tuned.plugins.plugin_bootloader: cannot update grubenv: ''
      2023-12-01 11:07:09,743 INFO     tuned.daemon.daemon: static tuning from profile 'cpu-partitioning' applied

       

      after reboot check kernel cmdline is right but the rx/tx queues of nic is incorrect.

      [root@dell-per750-64 ~]# cat /proc/cmdline 
      BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-528.rt7.317.el8.x86_64 root=/dev/mapper/rhel_dell--per750--64-root ro intel_iommu=on ksdevice=bootif crashkernel=auto resume=/dev/mapper/rhel_dell--per750--64-swap rd.lvm.lv=rhel_dell-per750-64/root rd.lvm.lv=rhel_dell-per750-64/swap console=ttyS0,115200n81 skew_tick=1 tsc=reliable rcupdate.rcu_normal_after_boot=1 isolcpus=managed_irq,domain,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111 intel_pstate=disable nosoftlockup nohz=on nohz_full=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111 rcu_nocbs=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111 irqaffinity=0,1
      [root@dell-per750-64 ~]# ethtool -l ens7f0
      Channel parameters for ens7f0:
      Pre-set maximums:
      RX:        112
      TX:        112
      Other:        1
      Combined:    112
      Current hardware settings:
      RX:        0
      TX:        0
      Other:        1
      Combined:    112
      

      Expected results

      tuned can take effect after reboot. the rx/tx queues will be adjusted to suit the number.

      Here is an output from rhel8.6.z rt kernel.

      # cat /proc/cmdline 
      BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-305.72.1.rt7.144.el8_4.x86_64 root=/dev/mapper/rhel_dell--per750--65-root ro intel_iommu=on ksdevice=bootif crashkernel=auto resume=/dev/mapper/rhel_dell--per750--65-swap rd.lvm.lv=rhel_dell-per750-65/root rd.lvm.lv=rhel_dell-per750-65/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,2-55,58-111 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=2-55,58-111 rcu_nocbs=2-55,58-111 irqaffinity=0,1,56,5
      [root@dell-per750-65 ~]# ethtool -l ens7f0
      Channel parameters for ens7f0:
      Pre-set maximums:
      RX:        112
      TX:        112
      Other:        1
      Combined:    112
      Current hardware settings:
      RX:        0
      TX:        0
      Other:        1
      Combined:    4
      

       

      Actual results

      tuned didn't work as well.

      Here is an original bug

      https://bugzilla.redhat.com/show_bug.cgi?id=1846767

              jskarvad Jaroslav Skarvada
              mhou@redhat.com Minxi Hou
              Jaroslav Skarvada Jaroslav Skarvada
              Robin Hack Robin Hack
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: