+++ This bug was initially created as a clone of Bug #1819666 +++

Description of problem:

Under some circumstances, tuned_params will duplicated show in kernel line. This may confuse people.

Version-Release number of selected component (if applicable):
4.18.0-193.rt13.51.el8.x86_64
tuned-2.13.0-6.el8.noarch

How reproducible:
100%

Steps to Reproduce:
1. Apply profile realtime-virtual-guest in guest

1. cat /etc/tuned/realtime-virtual-guest-variables.conf
   isolated_cores=1,2,3,4
   isolate_managed_irq=Y

1. tuned-adm profile realtime-virtual-guest

2. Reboot guest, the isolation info shows expected in kernel line.

1. cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.rt13.51.el8.x86_64 root=/dev/mapper/rhel_vm-73196-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm73-196-swap rd.lvm.lv=rhel_vm-73-196/root rd.lvm.lv=rhel_vm-73-196/swap skew_tick=1 isolcpus=managed_irq,domain,1,2,3,4 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,2,3,4 rcu_nocbs=1,2,3,4

3. Manually change kernel line. Append "default_hugepagesz=1G" to kernel line.

1. grubby --args="default_hugepagesz=1G" --update-kernel=`grubby --default-kernel`

4. Switch to another profile

1. tuned-adm profile desktop

5. Go back to profile realtime-virtual-guest again.

1. tuned-adm profile realtime-virtual-guest

6. Reboot RT guest

7. Check kernel line, the isolated info is duplicated show.

1. cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.rt13.51.el8.x86_64 root=/dev/mapper/rhel_vm-73196-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm73-196-swap rd.lvm.lv=rhel_vm-73-196/root rd.lvm.lv=rhel_vm-73-196/swap skew_tick=1 isolcpus=managed_irq,domain,1,2,3,4 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,2,3,4 rcu_nocbs=1,2,3,4 default_hugepagesz=1G skew_tick=1 isolcpus=managed_irq,domain,1,2,3,4 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,2,3,4 rcu_nocbs=1,2,3,4

Actual results:
Tuned_params is duplicated show in kernel line.

Expected results:
Tuned_params should only show one info of current profile.

Additional info:

— Additional comment from Jaroslav Škarvada on 2020-04-01 17:55:20 UTC —

Please try with
grubby-8.40-38.el8

and report results.

— Additional comment from Pei Zhang on 2020-04-02 07:06:31 UTC —

(In reply to Jaroslav Škarvada from comment #1)
> Please try with
> grubby-8.40-38.el8
>
> and report results.

Hi Jaroslav,

We were testing with grubby-8.40-38.el8.x86_64 in above testing. So with grubby-8.40-38.el8, we can hit this duplicate issue.

Best regards,

Pei

— Additional comment from RHEL Program Management on 2020-09-30 14:11:27 UTC —

pm_ack is no longer used for this product. The flag has been reset.

See https://issues.redhat.com/browse/PTT-1821 for additional details or contact lmiksik@redhat.com if you have any questions.

— Additional comment from Pei Zhang on 2020-10-12 12:17:41 UTC —

This issue still exists with 8.3 testing. After appending or modifying kernel options (like default_hugepagesz=1G or default_hugepagesz=2M), then rebooting os will cause isolated info repeatedly show. Just like below.

1. cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-240.el8.x86_64 root=/dev/mapper/rhel_dell-per74002-root ro crashkernel=auto resume=/dev/mapper/rhel_dellper740-02-swap rd.lvm.lv=rhel_dell-per740-02/root rd.lvm.lv=rhel_dell-per740-02/swap console=ttyS0,115200n81 skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup iommu=pt intel_iommu=on pci=realloc skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup default_hugepagesz=1G skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup

Versions:
4.18.0-240.el8.x86_64
tuned-2.14.0-3.el8.noarch
grubby-8.40-41.el8.x86_64

— Additional comment from Luiz Capitulino on 2020-10-15 03:44:07 UTC —

Pei,

Can we assume this issue exists on 8.2.z and 8.4 as well?

— Additional comment from Pei Zhang on 2020-10-15 04:07:11 UTC —

(In reply to Luiz Capitulino from comment #5)
> Pei,
>
> Can we assume this issue exists on 8.2.z and 8.4 as well?

Luiz,

This issue exists on 8.2.z.

I don't start 8.4 testing yet, but I think it should exist on 8.4 as well.

Besides, rhel7.9 doesn't hit this issue.

Best regards,

Pei

— Additional comment from Marcelo Tosatti on 2021-01-25 17:37:02 UTC —

Pei,

I can debug this if you make a system available.

TIA.

— Additional comment from Marcelo Tosatti on 2021-01-25 17:38:20 UTC —

Jaroslav, i can debug this problem (unless you have something else in mind).

Let me know if you want me to take it.

— Additional comment from Jaroslav Škarvada on 2021-01-25 18:59:18 UTC —

(In reply to Marcelo Tosatti from comment #8)
> Jaroslav, i can debug this problem (unless you have something else in mind).
>
> Let me know if you want me to take it.

Feel free to take it, but I think this could be already fixed upstream and in RHEL-8.4.0. Could you try: tuned-2.15.0-1.el8?

— Additional comment from Marcelo Tosatti on 2021-01-25 21:07:04 UTC —

(In reply to Jaroslav Škarvada from comment #9)
> (In reply to Marcelo Tosatti from comment #8)
> > Jaroslav, i can debug this problem (unless you have something else in mind).
> >
> > Let me know if you want me to take it.
>
> Feel free to take it, but I think this could be already fixed upstream and
> in RHEL-8.4.0. Could you try: tuned-2.15.0-1.el8?

Pei,

Perhaps try this tuned version and if it fails leave the machine available?

TIA

— Additional comment from Pei Zhang on 2021-01-26 05:10:34 UTC —

(In reply to Marcelo Tosatti from comment #10)
> (In reply to Jaroslav Škarvada from comment #9)
> > (In reply to Marcelo Tosatti from comment #8)
> > > Jaroslav, i can debug this problem (unless you have something else in mind).
> > >
> > > Let me know if you want me to take it.
> >
> > Feel free to take it, but I think this could be already fixed upstream and
> > in RHEL-8.4.0. Could you try: tuned-2.15.0-1.el8?
>
> Pei,
>
> Perhaps try this tuned version and if it fails leave the machine available?

Hello Marcelo,

This issue still exists with tuned-2.15.0-1.el8.noarch.

Here is the setup:

dell-per430-11.lab.eng.pek2.redhat.com root/kvmautotest

Step to reproduce (changing any of the kernel line options, then reboot, the tuned info will repeat again):

1. # python3 /root/reproduce.py

Best regards,

Pei

>
> TIA

— Additional comment from Jaroslav Škarvada on 2021-01-26 13:54:26 UTC —

(In reply to Pei Zhang from comment #11)
> (In reply to Marcelo Tosatti from comment #10)
> > (In reply to Jaroslav Škarvada from comment #9)
> > > (In reply to Marcelo Tosatti from comment #8)
> > > > Jaroslav, i can debug this problem (unless you have something else in mind).
> > > >
> > > > Let me know if you want me to take it.
> > >
> > > Feel free to take it, but I think this could be already fixed upstream and
> > > in RHEL-8.4.0. Could you try: tuned-2.15.0-1.el8?
> >
> > Pei,
> >
> > Perhaps try this tuned version and if it fails leave the machine available?
>
> Hello Marcelo,
>
> This issue still exists with tuned-2.15.0-1.el8.noarch.
>
> Here is the setup:
>
> dell-per430-11.lab.eng.pek2.redhat.com root/kvmautotest
>
> Step to reproduce (changing any of the kernel line options, then reboot, the
> tuned info will repeat again):
>
> 1. # python3 /root/reproduce.py
>
>
> Best regards,
>
> Pei
>
> >
> > TIA

I quickly checked the reproducer. It seems your machine has 'expanded' parameter 'option' in BLS entry:

1. cat /boot/loader/entries/*.x86_64.conf
   title Red Hat Enterprise Linux (4.18.0-277.el8.x86_64) 8.4 (Ootpa)
   version 4.18.0-277.el8.x86_64
   linux /vmlinuz-4.18.0-277.el8.x86_64
   initrd /initramfs-4.18.0-277.el8.x86_64.img $tuned_initrd
   options root=/dev/mapper/rhel_dell-per43011-root ro crashkernel=auto resume=/dev/mapper/rhel_dellper430-11-swap rd.lvm.lv=rhel_dell-per430-11/root rd.lvm.lv=rhel_dell-per430-11/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=1,3,5,7,9,11,13,15,17,19 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,3,5,7,9,11,13,15,17,19 rcu_nocbs=1,3,5,7,9,11,13,15,17,19 iommu=pt intel_iommu=on skew_tick=1 isolcpus=1,3,5,7,9,11,13,15,17,19 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,3,5,7,9,11,13,15,17,19 rcu_nocbs=1,3,5,7,9,11,13,15,17,19 skew_tick=1 isolcpus=1,3,5,7,9,11,13,15,17,19 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,3,5,7,9,11,13,15,17,19 rcu_nocbs=1,3,5,7,9,11,13,15,17,19 default_hugepagesz=1G $tuned_params
   ...

Notice 'options', it should be:
options $kernelopts $tuned_params

On cleanly provisioned machine just:
options $kernelopts

Because $kernelopts and $tuned_params are specified in grubenv. It seems grubby works the way that it 'expands' and 'flattens' the options.

— Additional comment from Jaroslav Škarvada on 2021-01-26 13:59:45 UTC —

I think grubby could (and should be extended) to correctly handle such cases and not flatten your 'hierarchic' grubenv.

I am proposing the following grubby changes:
a) when adding arguments, add it just to the 'options' in the BLS entry, do not expand other variables
b) when removing arguments, recursively go through the 'options' and all grubenv environment variables specified there and remove the arguments from the specific places (e.g. from the options or grubenv), do not 'flatten' the options.

— Additional comment from Pei Zhang on 2021-01-28 08:35:01 UTC —

(In reply to Jaroslav Škarvada from comment #13)
> I think grubby could (and should be extended) to correctly handle such cases
> and not flatten your 'hierarchic' grubenv.
>
> I am proposing the following grubby changes:
> a) when adding arguments, add it just to the 'options' in the BLS entry, do
> not expand other variables
> b) when removing arguments, recursively go through the 'options' and all
> grubenv environment variables specified there and remove the arguments from
> the specific places (e.g. from the options or grubenv), do not 'flatten' the
> options.

Hello Jaroslav,

Do you mean this issue should be on grubby component? Thanks.

Best regards,

Pei

— Additional comment from Jaroslav Škarvada on 2021-03-09 17:01:34 UTC —

Yes, it's grubby, I am reassigning this to grubby.

I think grubby could (and should be extended) to correctly handle such cases and not flatten your 'hierarchic' grubenv.

I am proposing the following grubby changes:
a) when adding arguments, add it just to the 'options' in the BLS entry, do not expand other variables
b) when removing arguments, recursively go through the 'options' and all grubenv environment variables specified there and remove the arguments from the specific places (e.g. from the options or grubenv), do not 'flatten' the options.

Or simple workaround: just ignore the $tuned_params and not expand them.

— Additional comment from Luiz Capitulino on 2021-03-12 01:01:56 UTC —

Grubby people,

It seems that bug 1777874 depends on this bug and bug 1777874
has a number of customer cases attached to it. I think this BZ
should be considered high priority.

— Additional comment from Pei Zhang on 2021-03-12 02:12:46 UTC —

Testing update:

This issue still exists with latest rhel8.4.

Versions:
grubby-8.40-41.el8.x86_64
4.18.0-296.el8.x86_64
tuned-2.15.0-2.el8.noarch

1. cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-296.el8.x86_64 root=/dev/mapper/rhel_dell-per74002-root ro crashkernel=auto resume=/dev/mapper/rhel_dellper740-02-swap rd.lvm.lv=rhel_dell-per740-02/root rd.lvm.lv=rhel_dell-per740-02/swap console=ttyS0,115200n81 skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=ffffffff,000002ab intel_pstate=disable nosoftlockup iommu=pt intel_iommu=on pci=realloc skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=ffffffff,000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=ffffffff,000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=ffffffff,000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=ffffffff,000002ab intel_pstate=disable nosoftlockup default_hugepagesz=1G skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=ffffffff,000002ab intel_pstate=disable nosoftlockup

— Additional comment from Pei Zhang on 2021-06-03 04:02:54 UTC —

Hello Jared,

Could you have a look about this bug? Do we plan to fix it on rhel8.5? Thanks a lot.

Best regards,

Pei

— Additional comment from Pei Zhang on 2021-06-03 04:06:35 UTC —

Testing update:
This issue still exists in rhel8.5.

Version:
grubby-8.40-41.el8.x86_64

BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-310.rt7.91.el8.x86_64 root=/dev/mapper/rhel_dell-per43010-root ro crashkernel=auto resume=/dev/mapper/rhel_dellper430-10-swap rd.lvm.lv=rhel_dell-per430-10/root rd.lvm.lv=rhel_dell-per430-10/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,1,3,5,7,9,11,13,15,17,19,12,14,16,18 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,3,5,7,9,11,13,15,17,19,12,14,16,18 rcu_nocbs=1,3,5,7,9,11,13,15,17,19,12,14,16,18 iommu=pt intel_iommu=on default_hugepagesz=1G skew_tick=1 isolcpus=managed_irq,domain,1,3,5,7,9,11,13,15,17,19,12,14,16,18 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,3,5,7,9,11,13,15,17,19,12,14,16,18 rcu_nocbs=1,3,5,7,9,11,13,15,17,19,12,14,16,18

BOOT_IMAGE=(hd0,msdos2)/vmlinuz-4.18.0-309.el8.x86_64 root=/dev/mapper/rhel_dell-per74002-root ro crashkernel=auto resume=/dev/mapper/rhel_dellper740-02-swap rd.lvm.lv=rhel_dell-per740-02/root rd.lvm.lv=rhel_dell-per740-02/swap console=ttyS0,115200n81 skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup iommu=pt intel_iommu=on pci=realloc skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11 tuned.non_isolcpus=000002ab intel_pstate=disable nosoftlockup default_hugepagesz=1G

— Additional comment from Petr Janda on 2021-06-04 11:18:49 UTC —

Agree with Jaroslav, the issue here is expanding variables during editing options line in /boot/loader/entries/*.conf by grubby

It can be reproduced completely without tuned and reboots

• install rhel-8.5 (RHEL-8.5.0-20210604.n.0, x86_64, grubby-8.40-41.el8.x86_64 in my case)

• list options line in bootentry file

1. grep ^options /boot/loader/entries/*el8.x86_64.conf
   options $kernelopts $tuned_params

• add a variable parameter

1. grubby --args='$myenv' --update-kernel=$(grubby --default-kernel)

• list options line in bootentry file again

1. grep ^options /boot/loader/entries/*el8.x86_64.conf
   options expanded_kernelopts_here $tuned_params $myenv

• define myenv in grubenv

1. grub2-editenv - set 'myenv=somename=somevalue'

• check grubenv

1. grub2-editenv list

• try to add $myenv options again

1. grubby --args='$myenv' --update-kernel=$(grubby --default-kernel)

• list options line in bootentry file again

1. grep ^options /boot/loader/entries/*el8.x86_64.conf
   options expanded_kernelopts_here $tuned_params expanded_myenv_here $myenv

qa_ack+

— Additional comment from Petr Janda on 2021-06-07 06:50:36 UTC —

I'd rather move discussion here, although IRC is much faster.

The reason for expanding variables during command line arguments update is to allow single menuentry to diverge from one stored in grubenv, but tuned doesn't expect it.

possible solutions
1) expand just $kernelopts variable and let other vars untouched
2) expand all variables except $tuned_params
3) include some logic and expand a variable only in the case a modified/removed argument is stored in it
4) do not use variables at all

Every solution has its drawbacks. Although 2) is probably enough for this specific bug, I would prefer some more robust solution.
As time for fixing it is quite short for some invasive change, I'm OK with it, but would propose to file a RFE for some more systematic solution.

— Additional comment from Javier Martinez Canillas on 2021-06-07 06:58:54 UTC —

(In reply to Petr Janda from comment #21)
> I'd rather move discussion here, although IRC is much faster.
>

Yes, agreed.

> The reason for expanding variables during command line arguments update is
> to allow single menuentry to diverge from one stored in grubenv, but tuned
> doesn't expect it.
>
> possible solutions
> 1) expand just $kernelopts variable and let other vars untouched
> 2) expand all variables except $tuned_params
> 3) include some logic and expand a variable only in the case a
> modified/removed argument is stored in it
> 4) do not use variables at all
>

For F34 and RHEL9 onwards we chose option (4). Using environment variables for
the kernel command line parameters was proven to be fragile and error prone,
so we just dropped the $kernelopts var and the cmdline is stored in the BLS
snippets themselves.

We still left the ability to use env vars in the "options" field but it is
discouraged. So I believe tuned should do the same for Fedora/RHEL9, and maybe
for RHEL8 too which would solve this issue as well.

> Every solution has its drawbacks. Although 2) is probably enough for this
> specific bug, I would prefer some more robust solution.

Yes, I'm OK with going with option (1). That is, to only make grubby to expand
the $kernelopts and leave other variables untouched. That means thought that a
single kernel entry wouldn't be able to have different tuned params than others
but I guess that's not a supported use case anyways ?

> As time for fixing it is quite short for some invasive change, I'm OK with
> it, but would propose to file a RFE for some more systematic solution.

I think both (1) and (2) are no invasive and I'm OK with going either way.

— Additional comment from Pei Zhang on 2021-06-15 14:00:04 UTC —

Thank you Javier, Petr, Jared for the efforts to fix this issue.

Testing update:

This issue has gone with grubby-8.40-42.el8.x86_64. No duplication tuned info any more.

1. cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-312.el8.x86_64 root=/dev/mapper/rhel_vm-73150-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm73-150-swap rd.lvm.lv=rhel_vm-73-150/root rd.lvm.lv=rhel_vm-73-150/swap skew_tick=1 nohz=on nohz_full=1 rcu_nocbs=1 tuned.non_isolcpus=0000003d intel_pstate=disable nosoftlockup iommu=pt intel_iommu=on default_hugepagesz=2M

1. cat /proc/cmdline
   BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-311.rt7.92.el8.x86_64 root=/dev/mapper/rhel_dell-per43011-root ro crashkernel=auto resume=/dev/mapper/rhel_dellper430-11-swap rd.lvm.lv=rhel_dell-per430-11/root rd.lvm.lv=rhel_dell-per430-11/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,1,3,5,7,9,11,13,15,17,19,12,14,16,18 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,3,5,7,9,11,13,15,17,19,12,14,16,18 rcu_nocbs=1,3,5,7,9,11,13,15,17,19,12,14,16,18 iommu=pt intel_iommu=on default_hugepagesz=2M

— Additional comment from Luiz Capitulino on 2021-06-16 20:16:23 UTC —

Pei,

Does this affect 8.4.z and maybe 8.2.z?

If yes, Javier, should we consider the backport? Those z-streams are EUS, they have a long lifecycle support for Telco companies (who could be affected by this issue).

— Additional comment from Javier Martinez Canillas on 2021-06-16 20:23:32 UTC —

(In reply to Luiz Capitulino from comment #24)
> Pei,
>
> Does this affect 8.4.z and maybe 8.2.z?
>
> If yes, Javier, should we consider the backport? Those z-streams are EUS,
> they have a long lifecycle support for Telco companies (who could be
> affected by this issue).

Yes, let's create z-stream clones for this.

— Additional comment from RHEL Program Management on 2021-06-16 20:23:38 UTC —

The BZ has been approved for cloning.

The BZ can be now clone by everyone with the self-service cloning tool http://watson.int.open.paas.redhat.com/rules

For more information regarding RHEL ZStream and cloning please visit https://docs.google.com/document/d/1yL8iTHjxyQ7sI-fC4PcPjpOOyfF5ECGnK-B7r_QRZm4/edit#

— Additional comment from RHEL Program Management Team on 2021-06-16 20:36:51 UTC —

This bug has been copied as 8.4.0 stream bug#1972908 and now must be resolved in the current update release, blocker flag set.

This bug has been copied as 8.2.0 stream bug#1972909 and now must be resolved in the current update release, blocker flag set.

— Additional comment from Pei Zhang on 2021-06-17 01:14:45 UTC —

(In reply to Luiz Capitulino from comment #24)
> Pei,
>
> Does this affect 8.4.z and maybe 8.2.z?

Luiz,

Yes, this affect both 8.4.z and 8.2.z. Thanks.

— Additional comment from Petr Zatko on 2021-08-12 11:28:02 UTC —

Reproduced using RHEL-8.5.0-20210609.n.3 with grubby-8.40-41
And
Tested using RHEL-8.5.0-20210810.t.16 with grubby-8.40-42

Testing was done by following comment 20

— Additional comment from RHEL Program Management on 2021-08-18 07:28:06 UTC —

The Current Deadline / ITM for this BZ has passed. Please discuss priorities and risk with your PO and Dev Contact, then revise the Current Deadline by updating the ITM.

More details about the deadline management are available at https://one.redhat.com/rhel-developer-guide/#_using_deadlines_to_prioritize_work

— Additional comment from errata-xmlrpc on 2021-08-18 13:50:49 UTC —

This bug has been added to advisory RHBA-2021:80376 by Jan Hlavac (jhlavac@redhat.com)

— Additional comment from errata-xmlrpc on 2021-08-18 13:50:51 UTC —

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2021:80376-01
https://errata.devel.redhat.com/advisory/80376

— Additional comment from errata-xmlrpc on 2021-08-18 13:51:03 UTC —

This bug has been added to advisory RHBA-2021:80376 by Jan Hlavac (jhlavac@redhat.com)

— Additional comment from Petr Janda on 2021-08-19 11:41:02 UTC —

Build was stuck in gating, moving ITM.

— Additional comment from Marta Lewandowska on 2021-08-25 07:41:38 UTC —

grubby-8.40-42.el8 is present in nightly compose RHEL-8.5.0-20210825.n.0
Moving to Verified.

— Additional comment from Red Hat Bugzilla on 2021-11-03 05:11:06 UTC —

remove performed by PnT Account Manager <pnt-expunge@redhat.com>

— Additional comment from Red Hat Bugzilla on 2021-11-03 05:11:11 UTC —

remove performed by PnT Account Manager <pnt-expunge@redhat.com>

— Additional comment from errata-xmlrpc on 2021-11-09 01:02:36 UTC —

Bug report changed to RELEASE_PENDING status by Errata System.
Advisory RHBA-2021:80376-02 has been changed to PUSH_READY status.
https://errata.devel.redhat.com/advisory/80376

— Additional comment from errata-xmlrpc on 2021-11-09 20:05:06 UTC —

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (grubby bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4509

— Additional comment from Vincent S. Cojot on 2023-08-29 01:56:18 UTC —

Is it possible to know if this fix was backported to 8.4 TUS?
When I look at the main BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1819666

it shows a few clones:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1921577%2C1972908%2C1972909&list_id=13310255

The 8.4 clone is: https://bugzilla.redhat.com/show_bug.cgi?id=1972908

However, the clone for 8.4 was closed as 'stale' in 2021, as exhibited here:
https://bugzilla.redhat.com/show_bug.cgi?id=1972908#c2

if I look at a 16.2 undercloud with TUS and latest packages, I can see this:
[root@osp16d ~]# rpm -qa tuned*
tuned-2.20.0-1.3.20230614git850368d2.el8fdp.noarch
tuned-profiles-cpu-partitioning-2.20.0-1.3.20230614git850368d2.el8fdp.noarch

'tuned' is coming from the fastdatapath repo:
[root@osp16d ~]# yum info tuned
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - High Availability - Telecommunications Update Service (RPMs) 48 kB/s | 2.6 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - High Availability - Telecommunications Update Service (RPMs) 20 MB/s | 3.1 MB 00:00
Red Hat Enterprise Linux 8 for x86_64 - High Availability - Extended Update Support (RPMs) 53 kB/s | 2.6 kB 00:00
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 - Extended Update Support (RPMs) 84 kB/s | 2.9 kB 00:00
Advanced Virtualization for RHEL 8 x86_64 (RPMs) 68 kB/s | 2.6 kB 00:00
Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs) 70 kB/s | 2.3 kB 00:00
Installed Packages
Name : tuned
Version : 2.20.0
Release : 1.3.20230614git850368d2.el8fdp
Architecture : noarch
Size : 994 k
Source : tuned-2.20.0-1.3.20230614git850368d2.el8fdp.src.rpm
Repository : @System
From repo : fast-datapath-for-rhel-8-x86_64-rpms <============================

However, when I check the changelog of that package, I cannot find any reference to 1972908 or 1819666

[root@osp16d ~]# rpm -q tuned --changelog|egrep '(1972908|1819666)'
[root@osp16d ~]#

so my question still stands. in what 8.4 TUS package should the fix be present?

— Additional comment from Vincent S. Cojot on 2023-08-30 16:18:38 UTC —

Reopening

— Additional comment from Marta Lewandowska on 2023-09-01 11:38:13 UTC —

(In reply to Vincent S. Cojot from comment #40)
> Is it possible to know if this fix was backported to 8.4 TUS?

It looks like a build was done (grubby-8.40-41.el8_4.1) but that build was deleted for
some reason, so it was never released. If you/someone needs this for 8.4.z, please (re-)open
a bug for the backport.

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1650795