Uploaded image for project: 'Observability and Data Analysis Program'
  1. Observability and Data Analysis Program
  2. OBSDA-392

Allow specifying custom index pattern in Cluster Log Forwarder

XMLWordPrintable

    • Icon: Feature Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • Log Collection, PM Logging
    • None
    • False
    • None
    • False
    • Not Selected
    • 0% To Do, 0% In Progress, 100% Done
    • Backlog Refinement
    • Observability

      What is the nature and description of the request?

      We want to send logs from OCP to Elastic search to a specific index. We need to use additional tools like fluent/logstash for the log routing to different indexes.

       

      Why does the customer need this?

      We need to add additional component to the stack without this functionality. It increases the complexity of the log management. We were also able to specify the indexes in older versions of OCP.

       

      Use case:

      3 PROD clusters and 4 app on them We ned to push the logs to an external elasticsearch. We can do so, but all log will end up in the same index app-write / audit-write / infra-write. But some of the logs need to be archived for 30days and some for one year. Different retention policy cannot be set for a single index. It is also easier to manage access to the logs for different people if the logs are in different indexes.

      List any affected packages or components.

      Cluster Log Forwarder

      Note From Comment Below:
      The petition here is to define to send the infrastructure, application or audit logs to an specific indice name desired as it could be ocpcluster1-write, ocpcluster2-write, ocpcluster3-write. Then, in the "external" Elasticsearch could be managed perfectly the retention days, log access, user able to write indices, etc depending on the cluster from where the logs are coming from.

      The supported is for when enabled "parse: json" and for the JSON logs detecetd, then, it allows to define a part of the indice where it will be

      {app,infra,audit}

      -<structuredTypeKey|structuredTypeName>-xxxxx. NOTE: This should be only to be enabled for when log forwarding to an "external" Elasticsearch. This should be only for when logfowarding to an external Elasticsearch.

            jamparke@redhat.com Jamie Parker
            rhn-support-rhodain1 Roman Hodain
            Votes:
            13 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated: