Loading...

XML

Word

Printable

Type: Bug
Resolution: Done-Errata
Priority: Critical
Fix Version/s: Logging 5.7.6
Affects Version/s: Logging 5.5.z, Logging 5.6.z
Component/s: Log Collection
Labels:
- devel_ack+
- v0.28
- vector

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
NEW
Release Note Text:

Hide
Before this change, the collector relied upon the default config setting when reading container log lines. This resulted in the collector not efficiently reading rotated files an high volume clusters and holding onto deleted file handles for a long time. This change increases the number of bytes read allowing the collector to more efficiently process rotated files.

Show
Before this change, the collector relied upon the default config setting when reading container log lines. This resulted in the collector not efficiently reading rotated files an high volume clusters and holding onto deleted file handles for a long time. This change increases the number of bytes read allowing the collector to more efficiently process rotated files.
Release Note Type:
Bug Fix
Release Note Status:
Proposed
Intelligence Requested:
Market:

Severity:
Critical

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

Modify the max_read_bytes of vector to allow it to more efficiently read rotated files to mitigate ~~LOG-3949~~.

Disk usage was consistently filling up, following one application pod around the environment. du did not show the disk usage, but lsof showed a large number of deleted files were still being locked by Vector:

vector 3430171 root 163r REG 8,4 105040954 1040189142 /var/log/pods/example-dev_example-cmd-linux-2_a9a87c45-ecad-49af-bdb7-3877273e5b95/example-cmd-linux-pod/0.log.20230403-205041 (deleted)

Deleting the collector pod (or killing the vector process) releases the files and they fully delete, clearing the space.

Version-Release number of selected component (if applicable):

cluster-logging.5.5.4

How reproducible:

So far failed to reproduce. At this time the application which caused the issue is no longer running so not currently able to gather data from original cluster as the issue is active.