Goals

Collector can be configured to listen for connections as an rsyslog server.

Logs from inbound connection can be normalized, filtered and forwarded using existingn features of the collector.

Non Goals

Once we can consume rsyslog logs, there may be need for new input filters or normalization features motivated by the structure of rsyslog records. This is not part of this Epic but is likely as future work.

Motivation

Requests to collect logs from outside the cluster using a common framework and log store.

In particular the OpenStack project has this need.

It is common for non-cluster applications to use syslog-based libraries for logging.

Rsyslog allows  us to collect such logs with minimal intrusion into the source applications.

Alternatives

Require external log sources to connect to LokiStack directly.

Don't provide a common log stores for cluster and non-cluster logs.

Acceptance Criteria

• Rsyslog logs received by the collector can be stored in Loki
• Rsyslog logs received by the collector can be forwarded to other output types.

Risk and Assumptions

Documentation Considerations

Open Questions

Distributing load from external sources

Currently  we run a collector per cluster node, which partitiions the total work of collecting from the cluster.

Data from outside the cluster isn't naturally partitioned this way, consider load-balancing and/or starting additional collector instances to handle external traffic.

Rsyslog round trip

We can already forward to rsyslog, question: if we receive from rsyslog and forward to rsyslog, can we preserve the log data exactly? Not clear if this is a requirement, in fact it may be desirable to add a JSON envelope on the inbound side and preserve it on the outbound side so messages from separate sources can be separated.

Additional Notes