Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5821

Loki configuration fails validation when retention is enabled, leading to crash loop

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Before this update, Loki Operator produced an invalid configuration when retention was enabled, causing Loki to not start up properly. With this update, the missing option is added to the configuration created by Loki Operator.
    • Bug Fix
    • Log Storage - Sprint 256
    • Critical

      Description of problem:

      After upgrading to Logging 5.8.9 from 5.8.8 makes Loki fails with the error:

       compactor.delete-request-store should be configured when retention is enabled

      all the pods are in crashloopback:

      $ oc get pods |grep -i loki
logging-loki-compactor-0                      0/1     CrashLoopBackOff   3 (9s ago)     62s
logging-loki-distributor-8d877484b-w24pk      0/1     CrashLoopBackOff   3 (7s ago)     62s
logging-loki-gateway-7b45878585-d2jw8         2/2     Running            0              61s
logging-loki-gateway-7b45878585-ffd5m         2/2     Running            1 (52s ago)    60s
logging-loki-index-gateway-0                  0/1     CrashLoopBackOff   2 (30s ago)    61s
logging-loki-ingester-0                       0/1     CrashLoopBackOff   2 (29s ago)    62s
logging-loki-querier-8545b64f7b-wwd2s         0/1     CrashLoopBackOff   3 (6s ago)     62s
logging-loki-query-frontend-79bb44959-ljcg8   0/1     CrashLoopBackOff   3 (3s ago)     62s

      Version-Release number of selected component (if applicable):

      Loki 5.8.9

      How reproducible:

      Always when retention is defined

      Steps to Reproduce:

      1. Have installed Loki 5.8.9
      2. Set retention in Loki:
        spec:
    limits:
      global:
        retention:
          days: 1
      tenants:
        application:
          retention:
            days: 30
            streams:
            - days: 30
              priority: 1
              selector: '{log_type="application"}'
        infrastructure:
          retention:
            days: 15
            streams:
            - days: 15
              priority: 1
              selector: '{log_type="infrastructure"}'

      Observe Loki failing with the error:

       compactor.delete-request-store should be configured when retention is enabled

      Actual results:

      Loki compactor in crashloopback

      Expected results:

      Loki pods working and able to set retention

      Additional info:

      Article https://access.redhat.com/solutions/7079499 for workarounds

              rojacob@redhat.com Robert Jacob
              rhn-support-ocasalsa Oscar Casal Sanchez
              Qiaoling Tang Qiaoling Tang
              Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

                Created:
                Updated:
                Resolved: