Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-684

Cluster LogForwarder Loki support

    XMLWordPrintable

Details

    • Log Forwarding: Loki
    • 5
    • OBSDA-7Adopting Loki as an alternative to Elasticsearch to support more lightweight, easier to manage/operate storage scenarios
    • Passed
    • px-ack
    • Green
    • Hide
      * With this update, you can forward log data to Grafana Loki, a horizontally-scalable, highly-available, multi-tenant log aggregation system. For more information, see xref:../logging/cluster-logging-external.html#cluster-logging-collector-log-forward-loki_cluster-logging-external[Forwarding logs to Grafana Loki]. (link:https://issues.redhat.com/browse/LOG-684[LOG-684])
      Show
      * With this update, you can forward log data to Grafana Loki, a horizontally-scalable, highly-available, multi-tenant log aggregation system. For more information, see xref:../logging/cluster-logging-external.html#cluster-logging-collector-log-forward-loki_cluster-logging-external[Forwarding logs to Grafana Loki]. (link: https://issues.redhat.com/browse/LOG-684 [ LOG-684 ])

    Description

      Goals

      • Provide a new output option to forward any logs to an existing Loki cluster with the following configuration options:
        • Tenant as endpoint URI path. One tenant per OutputRef
        • Server-side TLS support only
        • Translation of fluentd fields to Loki labels..

      Non-Goals

      • Changing or adopting the new Loki output into our "DEFAULT" pipeline.
      • Us supporting Loki.
      • No custom field translation
      • No basic authentication

      Motivation

      Loki is becoming a hot topic pretty quick. We already have a handful (and more) customers that are really interested in adopting it themselves and we also plan to adopt it as an alternative to our current storage engine - the Elastic stack. We need to make sure that we let users easily integrate Loki into our log pipeline as well as our ability to switch the "DEFAULT" pipeline to Loki in the future.

      Alternatives

      Acceptance Criteria

      • Verify that any log messages from a particular input source (infra, app, and audit) are forwarded and present inside a remote Loki (can run locally, but the important part is that we standup a standalone Loki cluster).
      • Verify that we translate Loki configuration into a valid Fluentd config.
      • Verify that instead of using fields for docker., kubernetes., pipeline_metadata.*, we translate them into labels.

      Risk and Assumptions

      Documentation Considerations

      • Documentation describing what is (is not ) configurable.

      Open Questions

      Additional Notes

      • With the new log forwarding API, we are now have the ability to introduce other endpoints to which we forward logs without compromising the supportability of OpenShift Logging.
      • Loki is another promising endpoint

      Attachments

        Issue Links

          Activity

            People

              rhn-engineering-aconway Alan Conway
              openshift_jira_bot OpenShift Jira Bot
              Kabir Bharti Kabir Bharti
              Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: