Story

A log batch is a collection of log records that are written to the remote target in a single action (single HTTP request, message, socket write or whatever)

As a log admin

• I want to limit the maximum size of a batch of log records written to an ouput.
• For outputs that have limts on batch size by default, I want the collector to enforce the correct batch size by default.

Design

LOG-2207 and LOG-2298 introduce a limit map field to forwarder outputs, with a maxRecordsPerSecond sub-field. This issue adds two new sub-fields to limit:

• maxBatchBytes: The maximum size for a batch of records to be written to a remote log processor.

If maxBatchBytes is absent, then there may or may not be a limit. The forwarder should choose a default appropriate for the output type.

Records will not be truncated as long as at least one record can fit in the batch.

If a single record is larger than maxBatchBytes, then it will be truncated as described in LOG-2758.

Acceptance Criteria

• Batch limits are enforced as described, tested for at least: loki, syslog, cloudwatch outputs.
• Default batch limits are enforced, tested for at least: loki, cloudwatch. Sending a large log file should not cause errors like this:

(500 Internal Server Error rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6258939 vs. 4194304) 

NOTE: see the related issue links for issues reported  against specific output types.