Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5616

[release-5.7] Replay Memory Ceiling not set when 1x.extra-small size is used

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      Before this update, the Loki Operator configured Loki with a write-ahead log replay memory ceiling of zero bytes when using the "1x.extra-small" size causing delays when restarting ingesters. With this update, the minimum value used for the replay memory ceiling has been increased to a sensible value.
      Show
      Before this update, the Loki Operator configured Loki with a write-ahead log replay memory ceiling of zero bytes when using the "1x.extra-small" size causing delays when restarting ingesters. With this update, the minimum value used for the replay memory ceiling has been increased to a sensible value.
    • Bug Fix
    • Log Storage - Sprint 254

      Description of problem:

      When a LokiStack is configured as "1x.demo", the generated Loki configuration contains a "replay_memory_ceiling" value of zero. This value is respected by the ingester and leads to it immediately flushing each chunk during WAL-replay, which slows down ingester startup times considerably.

      Version-Release number of selected component (if applicable):

      5.9.2

      Steps to Reproduce:

      1. Create a LokiStack configured with "size: 1x.demo"
      2. Look into the generated ConfigMap for the "replay_memory_ceiling" value

      Actual results:

      "replay_memory_ceiling" value is zero.

      Expected results:

      Use a more sensible value for the replay_memory_ceiling, while keeping the usage low for demo installations.

      Additional info:

      When this behavior of the ingester occurs it looks as if the ingester is "endlessly looping" on flushing the logs, because almost each line is flushed to object storage on its own.

      It is no endless loop though, it is just very slow. Flushing after a restart can take a few minutes even on small clusters when this bug is present. Once the fix was applied, the flush after restart only took a few seconds, if at all.

            rojacob@redhat.com Robert Jacob
            rojacob@redhat.com Robert Jacob
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: