Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30598

fs.inotify.max_user_watches too low

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Normal
    • None
    • 4.13.z, 4.14.z, 4.15.z, 4.16.0
    • No
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      We deploy /etc/sysctl.d/inotify.conf which was meant to increase the number of watches to 65k over the 8k default present in RHEL8. However RHEL9 kernel moved to a model where the value scales based on the amount of memory present in the system using up to 1% of addressable memory.
      
      See https://github.com/torvalds/linux/commit/92890123749bafc317bbfacbe0a62ce08d78efb7
      and 
      https://github.com/openshift/machine-config-operator/pull/2944#issuecomment-1030236808
      
      In RHCOS9 now it's no longer necessary and the default provides better scalability

      Version-Release number of selected component (if applicable):

      4.13.z, 4.14.z, 4.15.z, 4.16.0

      How reproducible:

      Install 4.13 or later

      Steps to Reproduce:

          1. sysctl fs.inotify.max_user_watches
          2. 
          3.
          

      Actual results:

      observe 65k value independent of instance size

      Expected results:

      value scaled based on memory up to 1048576 on instances with 128GiB or more

      Additional info:

      I did a quick test on small instances down to 4GiB to ensure that we don't scale down below the 65k value in supported instances, at least in my quick test that didn't happen.
      
      We may also want to consult with the Node Tuning Operator folks as it looks like the previous attempt to remove this file may've interacted with them.
      
      Finally, I guess since we still support RHEL Package Based workers (formerly known as RHEL workers) we likely need to find some way to ensure that we still apply these values in RHEL8. I don't know how to do that via templates. 
      
      Apparently other platforms like EKS have a value of 120,000.  

      CC jmencak 

      Attachments

        Activity

          People

            team-mco Team MCO
            rhn-support-sdodson Scott Dodson
            Sergio Regidor de la Rosa Sergio Regidor de la Rosa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: