-
Bug
-
Resolution: Obsolete
-
Undefined
-
None
-
4.14.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
3
-
None
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
can't use reliably a tuned profile containing vm.dirty_bytes or vm.dirty_background_bytes
Version-Release number of selected component (if applicable):
OCP 4.14.26
How reproducible:
randomly
Steps to Reproduce:
1. load a profile in this fashion
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
name: nce-tuned
namespace: openshift-cluster-node-tuning-operator
spec:
profile:
- data: |
[main]
summary=Change dirty memory
include=openshift-node
[sysctl]
vm.dirty_background_bytes = 104857600
vm.dirty_bytes = 419430400
name: openshift-sysctl-dirty
- data: |
[main]
summary=Change dirty memory
include=openshift-control-plane
[sysctl]
vm.dirty_background_bytes = 104857600
vm.dirty_bytes = 419430400
name: openshift-sysctl-ctlplane-dirty
recommend:
- match:
- label: node-role.kubernetes.io/master
priority: 10
profile: openshift-sysctl-ctlplane-dirty
- match:
- label: node-role.kubernetes.io/worker
priority: 20
profile: openshift-sysctl-dirty
2. oc get profile -n openshift-cluster-node-tuning-operator
3. oc logs -n openshift-cluster-node-tuning-operator -l openshift-app=tuned --tail=1000 | grep -i err
Actual results:
some profile are degraded
some errors:
ERROR tuned.plugins.plugin_sysctl: Failed to set sysctl parameter 'vm.dirty_background_bytes' to '0': [Errno 22] Invalid argument
ERROR tuned.plugins.plugin_sysctl: Failed to set sysctl parameter 'vm.dirty_bytes' to '0': [Errno 22] Invalid argument
Expected results:
profile not degraded
Additional info:
there are 2 ways to set the dirty memory, the default is to use dirty_ratio and not dirty_bytes.
When setting dirty_bytes, the dirty_ratio moves to 0 (meaning, "deactivated"). When setting dirty_ratio to 0, dirty_bytes moves to 0. So the right way for tuned to fallback a change of dirty_ratio is to set back the previous value of dirty_ratio or dirty_bytes, which ever is non 0
Here is a way to reproduce with tuned alone. This will log the error in tuned logs but it won't fail (tuned-2.22.1-1.el9.noarch). Create profiles:
cat > /etc/tuned/frigo/tuned.conf << EOF [main] summary=Customize dirty memory [sysctl] vm.dirty_background_bytes = 104857600 vm.dirty_bytes = 419430400 EOF cat > /etc/tuned/frigo2/tuned.conf << EOF [main] summary=Customize dirty memory [sysctl] EOF
then select the empty profile with tuned-adm profile frigo2
Reboot the VM, then move to the other profile, and back
tuned-adm profile frigo
tuned-adm profile frigo2
then check the logs
$ tail /var/log/tuned/tuned.log 2024-06-03 09:18:52,619 INFO tuned.daemon.daemon: starting tuning 2024-06-03 09:18:52,620 INFO tuned.plugins.plugin_sysctl: reapplying system sysctl 2024-06-03 09:18:52,620 INFO tuned.daemon.daemon: static tuning from profile 'frigo' applied 2024-06-03 09:18:54,336 INFO tuned.daemon.daemon: stopping tuning 2024-06-03 09:18:54,337 ERROR tuned.plugins.plugin_sysctl: Failed to set sysctl parameter 'vm.dirty_background_bytes' to '0': [Errno 22] Invalid argument 2024-06-03 09:18:54,337 ERROR tuned.plugins.plugin_sysctl: Failed to set sysctl parameter 'vm.dirty_bytes' to '0': [Errno 22] Invalid argument 2024-06-03 09:18:54,337 INFO tuned.profiles.loader: loading profile: frigo2 2024-06-03 09:18:54,337 INFO tuned.daemon.daemon: starting tuning 2024-06-03 09:18:54,338 INFO tuned.plugins.plugin_sysctl: reapplying system sysctl 2024-06-03 09:18:54,339 INFO tuned.daemon.daemon: static tuning from profile 'frigo2' applied
the workaround is to use only dirty_ratio and dirty_background_ratio.
Unfortunately on system with a bazillion RAM, setting the minimum of 1 in dirty_ratio as for 1% of a bazillion still ends up with a ton of memory.
Beside these logs tuned does not show any failure, only the node tuning operator shows an apparent failure.
- depends on
-
-
- Closed
-
