Loading...

Type: Bug
Resolution: Done-Errata
Priority: Critical
Fix Version/s: Logging 5.7.7
Affects Version/s: Logging 5.5.z, Logging 5.6.z
Component/s: Log Collection
Labels:
- devel_ack+
- v0.28
- vector

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
ASSIGNED
Release Note Text:

Hide
Before this change, the collector relied upon the default config setting when reading container log lines. This resulted in the collector not efficiently reading rotated files an high volume clusters and holding onto deleted file handles for a long time. This change increases the number of bytes read allowing the collector to more efficiently process rotated files.

Show
Before this change, the collector relied upon the default config setting when reading container log lines. This resulted in the collector not efficiently reading rotated files an high volume clusters and holding onto deleted file handles for a long time. This change increases the number of bytes read allowing the collector to more efficiently process rotated files.
Release Note Type:
Bug Fix
Release Note Status:
Proposed
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 235, Log Collection - Sprint 238, Log Collection - Sprint 239, Log Collection - Sprint 240, Log Collection - Sprint 241, Log Collection - Sprint 242, Log Collection - Sprint 243
Severity:
Critical

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

Disk usage was consistently filling up, following one application pod around the environment. du did not show the disk usage, but lsof showed a large number of deleted files were still being locked by Vector:

vector 3430171 root 163r REG 8,4 105040954 1040189142 /var/log/pods/example-dev_example-cmd-linux-2_a9a87c45-ecad-49af-bdb7-3877273e5b95/example-cmd-linux-pod/0.log.20230403-205041 (deleted)

Deleting the collector pod (or killing the vector process) releases the files and they fully delete, clearing the space.

Version-Release number of selected component (if applicable):

cluster-logging.5.5.4

How reproducible:

So far failed to reproduce. At this time the application which caused the issue is no longer running so not currently able to gather data from original cluster as the issue is active.