Problem statement

The fluent collector is labelling metrics with the path of the log file, which creates too many time series and can overload prometheus.

Background

This used to be ok when CRIO used /var/log/containers and used a single file name for the life of the log file.

After switching to /var/log/pods, CRIO now generates a series of numbered files, with the latest logs at the highest number.

This means that including the file label multiples the number of time series by the number of file rotations, which overloads prometheus.

Solution

Remove the path label. It is no longer needed, and should have been removed when we switched to /var/log/pods