Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-75773

TuneD race when hotplugging block devices results in garbled TUNED_BOOT_CMDLINE

Linking RHIVOS CVEs to...Migration: Automation ...Sync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • rhel-9.6
    • rhel-9.4, rhel-9.4.z
    • tuned
    • None
    • tuned-2.25.1-1.el9
    • No
    • Important
    • Patch, Upstream
    • rhel-sst-cs-net-perf-services
    • rhel-sst-cs-net-perf-services
    • ssg_core_services
    • 26
    • 8
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None

      What were you trying to do that didn't work?

      Run add/remove block devices while TuneD is running.  Restart TuneD.  There is a rare race where the TUNED_BOOT_CMDLINE will be garbled in /etc/tuned/bootcmdline when we hit the race.

      What is the impact of this issue to you?

      Failure to set kernel parameters on the entire OCP cluster, because the TuneD agents running on the nodes do not agree on kernel parameters.

      Please provide the package NVR for which the bug is seen:

      tuned-2.24.0-2.1.20240819gitc082797f.el9fdp.noarch.rpm

      How reproducible is this bug?

      Rare, but we have a reproducer script which seems to reproduce the issue relatively reliably.

      Steps to reproduce

      devices=100
      
      create_block_devices() {
        for d in $(seq 1 $devices)
        do
          dmsetup create dummy$d --table '0 4092 zero'
          sleep 0.001s
        done
      }
      
      delete_block_devices() {
        for d in $(seq 1 $devices)
        do
          dmsetup remove dummy$d 2>/dev/null
        done
      }
      
      delete_block_devices
      
      for i in $(seq 1 100)
      do
        create_block_devices &
        timeout 4 /usr/sbin/tuned --no-dbus --log tuned$i.log &
        pid=$!
        wait $pid
        grep ^TUNED_BOOT_CMDLINE /etc/tuned/bootcmdline | tee -a bootcmdline.log
        delete_block_devices
        sed -i 's/TUNED_BOOT_CMDLINE=.*$/TUNED_BOOT_CMDLINE=""/' /etc/tuned/bootcmdline
      done
      

      TuneD profile

      [main]
      include=throughput-performance
      
      [bootloader]
      cmdline_pstate=intel_pstate=${f:intel_recommended_pstate}
      

      Expected results

      No race and consistently correct value of TUNED_BOOT_CMDLINE in /etc/tuned/bootcmdline.

      Actual results

      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE=">4096disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      TUNED_BOOT_CMDLINE="intel_pstate=disable"
      

      Additional context

              pzacik@redhat.com Pavol Zacik
              jmencak Jiri Mencak
              Jaroslav Škarvada Jaroslav Škarvada
              Robin Hack Robin Hack
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: