Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-28757

Network monitors executed even though Dynamic Tuning is disabled

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • None
    • rhel-9.2.0.z
    • tuned
    • None
    • Important
    • Patch, Upstream
    • rhel-sst-cs-net-perf-services
    • ssg_core_services
    • 3
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None

      What were you trying to do that didn't work?

      This bug was reported by a customer who was using Node Tuning Operator and rebooted the node https://issues.redhat.com/browse/OCPBUGS-29720. After the node reboot the tuned profile degraded and the logs indicate that there was issue with the network monitors even though the dynamic tuning is disabled:

      2024-02-19T12:54:37.089039432+00:00 stdout F Exception in thread Thread-1:2024-02-19T12:54:37.089086057+00:00 stdout F Traceback (most recent call last):2024-02-19T12:54:37.089105883+00:00 stdout F   File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner2024-02-19T12:54:37.089536989+00:00 stdout F     self.run()2024-02-19T12:54:37.089557652+00:00 stdout F   File "/usr/lib64/python3.9/threading.py", line 917, in run2024-02-19T12:54:37.089721026+00:00 stdout F     self._target(*self._args, **self._kwargs)2024-02-19T12:54:37.089736771+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/daemon/daemon.py", line 197, in _thread_code2024-02-19T12:54:37.089969898+00:00 stdout F     self._unit_manager.create(self._profile.units)2024-02-19T12:54:37.089992577+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/units/manager.py", line 111, in create2024-02-19T12:54:37.090214819+00:00 stdout F     instance.plugin.initialize_instance(instance)2024-02-19T12:54:37.090232618+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/plugins/base.py", line 122, in initialize_instance2024-02-19T12:54:37.090444903+00:00 stdout F     self._instance_init(instance)2024-02-19T12:54:37.090461593+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/plugins/plugin_net.py", line 181, in _instance_init2024-02-19T12:54:37.090980587+00:00 stdout F     instance._load_monitor = self._monitors_repository.create("net", instance.assigned_devices)2024-02-19T12:54:37.091009954+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/monitors/repository.py", line 27, in create2024-02-19T12:54:37.091098632+00:00 stdout F     monitor_instance = monitor_cls(devices)2024-02-19T12:54:37.091115278+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/monitors/base.py", line 85, in __init__2024-02-19T12:54:37.091221539+00:00 stdout F     self.update()2024-02-19T12:54:37.091241559+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/monitors/monitor_net.py", line 39, in update2024-02-19T12:54:37.091333867+00:00 stdout F     cls._updateStat(device)2024-02-19T12:54:37.091350502+00:00 stdout F   File "/usr/lib/python3.9/site-packages/tuned/monitors/monitor_net.py", line 33, in _updateStat2024-02-19T12:54:37.091366277+00:00 stdout F     with open("/sys/class/net/" + dev + "/statistics/" + f) as statfile:2024-02-19T12:54:37.091381558+00:00 stdout F FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/ens3f0v40/statistics/rx_bytes'2024-02-19T12:55:36.255869002+00:00 stderr F E0219 12:55:36.255828    7380 controller.go:1218] timeout (60) to apply TuneD profile; restarting TuneD daemon2024-02-19T12:55:36.256089634+00:00 stdout F 2024-02-19 12:55:36,255 INFO     tuned.daemon.controller: terminating controller2024-02-19T12:55:36.273103118+00:00 stderr F I0219 12:55:36.273081    7380 controller.go:573] starting tuned...2024-02-19T12:55:36.308447406+00:00 stderr F I0219 12:55:36.308421    7380 controller.go:494] extracting cloud provider name to /var/lib/tuned/provider2024-02-19T12:55:36.308580886+00:00 stderr F I0219 12:55:36.308555    7380 controller.go:1056] updated Profile master0.winterfell-mno-2.lab.neat.nsn-rdnet.net bootcmdline: 2024-02-19T12:55:36.308759890+00:00 stderr F I0219 12:55:36.308708    7380 controller.go:536] written "/etc/tuned/recommend.d/50-openshift.conf" to set TuneD profile rdpmc-patch-master2024-02-19T12:55:36.308814702+00:00 stderr F I0219 12:55:36.308799    7380 controller.go:1236] previous application of TuneD profile failed; change detected, scheduling full restart in 1s2024-02-19T12:55:36.370809839+00:00 stdout F 2024-02-19 12:55:36,370 INFO     tuned.daemon.application: TuneD: 2.21.0, kernel: 5.14.0-284.50.1.el9_2.x86_642024-02-19T12:55:36.370835845+00:00 stdout F 2024-02-19 12:55:36,370 INFO     tuned.daemon.application: dynamic tuning is globally disabled

      In the code it is visible that the dynamic tuning is disabled and yet the call to update_monitors is made. This called at the init stage inspite of the fact that the dynamic tuning is disabled. 

      Please provide the package NVR for which bug is seen:

      How reproducible:

      Steps to reproduce

      1. Ensure that /etc/tuned/tuned-main.conf is configured with dynamic_tuning = 0 (disabled by default since tuned version 2.22.0 )
      2. Run Tuned
      3. We notice that load_monitors are called at init stage (https://github.com/redhat-performance/tuned/blob/master/tuned/plugins/plugin_net.py#L177 )

      Expected results

      UpdateStats (https://github.com/redhat-performance/tuned/blob/v2.21.0/tuned/monitors/monitor_net.py#L30-L35 ) should not be called as dynamic tuning is disabled.

       

              jskarvad Jaroslav Skarvada
              swsehgal@redhat.com Swati Sehgal
              Jaroslav Skarvada Jaroslav Skarvada
              Robin Hack Robin Hack
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: