Description of problem:

When more fsnotify events are generated than fs.inotify.max_queued_events sysctl setting, LogFileMetricExporter receives ErrEventOverflow (wrapper over IN_Q_OVERFLOW) error from fsnotify Go library, but this error is not handled.

To handle this error means application's state is no longer in sync with the filesystem state. can no longer be sure of the filesystem activity. The only reasonable way to handle this error is to close the inotify file descriptor and recreate all the watches. In case of LogFileMetricExporter, the container can exit and recreate itself from kubernetes.

Version-Release number of selected component (if applicable): v1.1

How reproducible:

It is hard to reproduce on local machine. But it can be reproduced.

Steps to Reproduce:

1. reduce {{fs.inotify.max_queued_events }}.
2. create lots of files in directories

Actual results:

After the program receives this error, more file events do not get processed.

Expected results:

The watches should be recreated once this condition occurs

Additional info:

Reference:

Good article on inotify A deeper investigation of inotify