Uploaded image for project: 'Observability and Data Analysis Program'
  1. Observability and Data Analysis Program
  2. OBSDA-123

cluster logging - delete ES logs by PVC usage threshold

XMLWordPrintable

    • False
    • False
    • 0% To Do, 0% In Progress, 100% Done
    • 0

      Is there a way to delete old indices from the default log store / clean logs when the log store becomes full more than a certain threshold

      Customer is aware of the retentionPeriod that it can configure on the clusterlogging CR, and customer have set it to 7d:

      ~~~
      application:
      maxAge: 1d
      audit:
      maxAge: 7d
      infra:
      maxAge: 7d
      type: elasticsearch
      ~~~

      Customer is looking for a feature to delete logs from the elasticsearch-storage PVC in case the elasticsearch-storage PVC fills up before the specified retention period is over?

      Recommendations from the access portal when the ES volume fills up :

      1. Delete manually/scale ES replicas down and back up to delete logs from volume (refs:

      https://access.redhat.com/solutions/5963541

      ,

      https://access.redhat.com/solutions/5504441

      )
      2. Resize existing volume (ref:

      https://access.redhat.com/solutions/5504441

      ,

      https://access.redhat.com/solutions/5552261

      )

      However, these manual steps will become unrealistic to do as the number of clusters goes up.

      When does the behavior occur? Frequency? Repeatedly? At certain times?
      Occasionally, if the ES storage fills up

      What is the business impact? Please also provide timeframe information.
      1. Manual steps such as resizing the PV manually when it fills up or deleting logs from the PV manually, will become unrealistic to do as the number of clusters goes up
      2. If the PV fills up then ES will block and fluentd will stop sending logs to ES. This will mean fluentd will also block all other outputs

            ptsiraki@redhat.com Periklis Tsirakidis
            rhn-support-aharchin Akhil Harchinder (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: