Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2746

CloudWatch forwarding rejecting large log events, fills tmpfs

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • * Before this update, clusters configured to forward logs to Amazon CloudWatch wrote rejected log files to temporary storage, causing cluster instability over time. With this update, chunk backup for CloudWatch has been disabled, resolving the issue.
    • Logging (Core) - Sprint 220, Log Collection - Sprint 221
    • Critical

    Description

      On several clusters configured to perform CloudWatch forwarding, the following condition has been observed in collector containers:

      Log event in xxxxxx is discarded because it is too large: 301486 bytes exceeds limit of 262144 (Fluent::Plugin::CloudwatchLogsOutput::TooLargeEventError) 

      The rejected logs are written to tmpfs on the node running the collector pod:

      2022-05-16 22:46:15 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3fe9caf3da38/5df28c737506490be5e3e7426bc2648f.log 

      Over a sustained period of time, these logs eventually fill the available tmpfs space on the nodes, leading to memory exhaustion. If this occurs on the control plane nodes, it eventually brings the cluster into instability.

      For the two clusters we've observed this on, it was cluster audit logs that trigger the 'too large' warning.

      Creating this Jira on request per Slack thread [0], cc rhn-engineering-aconway 

      https://coreos.slack.com/archives/CB3HXM2QK/p1652711924055009?thread_ts=1652660899.213729&cid=CB3HXM2QK

      Attachments

        Issue Links

          Activity

            People

              jcantril@redhat.com Jeffrey Cantrill
              mbargenq Matt Bargenquast (Inactive)
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: