Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-82265

tuned-2.22.1-3.el9_4 is crashing mid profile activation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • rhel-9.4.z
    • rhel-9.4.z
    • tuned
    • tuned-2.22.1-3.el9_4.1
    • Yes
    • Moderate
    • rhel-sst-cs-net-perf-services
    • ssg_core_services
    • 5
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • Unspecified
    • Unspecified
    • Unspecified
    • None

      What were you trying to do that didn't work?

      MicroShift CI has a job testing low latency and it started failing recently.

      Here's last 4.18 job that was successful (successful means that oslat test didn't found any of the cores to have a too big of a latency).

      Here's the 4.18 job next day which failed - there were no changes on the microshift side, but I noticed that in the latter the tuned package was updated to tuned-2.22.1-3.el9_4.noarch from the EUS repository.

      There's also a 4.19 run that uses even newer tuned-2.25.1-1.1.20250203git889387b0.el9fdp.noarch from fast-datapath repository and it was successful.

       

      I verified my assumption manually and when using 2.22.1-3.el9_4 I see following error in /var/log/tuned/tuned.log:

      2025-03-05 13:17:16,382 INFO     tuned.plugins.plugin_systemd: setting 'CPUAffinity' to '0 1 2 3 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95' in the '/etc/systemd/system.conf'
2025-03-05 13:17:16,901 ERROR    tuned.units.manager: BUG: Unhandled exception in start_tuning: 'GlobalConfig' object has no attribute 'get_list'
2025-03-05 13:17:16,901 ERROR    tuned.units.manager: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tuned/units/manager.py", line 119, in _try_call
    return f(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/tuned/plugins/instance/instance.py", line 80, in apply_tuning
    self._plugin.instance_apply_tuning(self)
  File "/usr/lib/python3.9/site-packages/tuned/plugins/base.py", line 273, in instance_apply_tuning
    self._instance_apply_static(instance)
  File "/usr/lib/python3.9/site-packages/tuned/plugins/plugin_script.py", line 103, in _instance_apply_static
    self._call_scripts(instance._scripts, ["start"])
  File "/usr/lib/python3.9/site-packages/tuned/plugins/plugin_script.py", line 78, in _call_scripts
    if not self._safe_script_path(script):
  File "/usr/lib/python3.9/site-packages/tuned/plugins/base.py", line 217, in _safe_script_path
    profile_paths = self._global_cfg.get_list(consts.CFG_PROFILE_DIRS, consts.CFG_DEF_PROFILE_DIRS)
AttributeError: 'GlobalConfig' object has no attribute 'get_list'2025-03-05 13:17:16,901 INFO     tuned.plugins.plugin_bootloader: generating initrd image from directory '/tmp/tmp.CqJ8Xa9abd'
2025-03-05 13:17:16,908 INFO     tuned.plugins.plugin_bootloader: installing initrd image as '/boot/tuned-initrd.img'
2025-03-05 13:17:16,909 INFO     tuned.plugins.plugin_bootloader: removing directory '/tmp/tmp.CqJ8Xa9abd'
2025-03-05 13:17:16,910 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2025-03-05 13:17:16,931 INFO     tuned.daemon.daemon: static tuning from profile 'microshift-baseline' applied

      And this is what I see on tuned-2.25.1-1.1.20250203git889387b0.el9fdp and  tuned-2.22.1-1.el9:

      2025-03-05 13:16:04,053 INFO     tuned.plugins.plugin_systemd: setting 'CPUAffinity' to '0 1 2 3 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95' in the '/etc/systemd/system.conf'
2025-03-05 13:16:04,544 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/cpu-partitioning/script.sh' with arguments '['start']'
2025-03-05 13:16:04,584 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/cpu-partitioning/script.sh' error output: 'modinfo: ERROR: Module kvm not found.
modinfo: ERROR: Module kvm not found.
modinfo: ERROR: Module kvm_intel not found.'
2025-03-05 13:16:04,584 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/microshift-baseline/script.sh' with arguments '['start']'
2025-03-05 13:19:10,771 INFO     tuned.daemon.daemon: static tuning from profile 'microshift-baseline' applied

      What is the impact of this issue to you?

      Tuned crashes and doesn't execute script of microshift-baseline profile resulting in desired CPU cores not being offlined and thus ruining the test (because oslat container gets CPUs that are not isolated).

      Please provide the package NVR for which the bug is seen:

      tuned-2.22.1-3.el9_4.noarch

      How reproducible is this bug?:

      Everytime

      Steps to reproduce

      CI procedure we have in place

      Expected results

      No error, scripts run to completion

      Actual results

      TuneD crashes and profiles' scripts are not executed

              rh-ee-jskunda Jakub Skunda
              pmatusza@redhat.com Patryk Matuszak
              Jakub Skunda Jakub Skunda
              Suyash Balasaheb Nalawade Suyash Balasaheb Nalawade
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: