Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-8124

system-redis is accumulating a huge amount of keys (sidekiq-batch issues)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • System
    • False
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • No

      Investigate why system-sidekiq redis instance accumulates so much redis keys taking huge memory space when should be almost zero wehn 0 enqueued jobs.
      From time to time a cleanup need to be done by system team, removing processed sidekiq-batches with huge default TTL of 30 days, removing big keys related to billing that are never deleted… (I remember some recent change from Hery removing records older than 24hours)

      February 3rd, 2023:
      It's confirmed that the accumulation of the keys is due to sidekiq-batch. For some reason, the keys are not being deleted, while they are expected to, once the batch is completed. There is a PR (closed without merging) with some code to clean up stale BID-* keys: https://github.com/3scale/porta/pull/2586
      This can be run periodically to clean up the System Redis, until the proper fix is applied.

      Also, some data in the sidekiq-batch entries does not seem to be correct (negative values for task children, etc), this also needs to be investigated.

      It is interesting, that in *Staging* the accumulation apparently is not reproduced, the Redis memory usage is low compared to Production, this also needs to be investigated. On the othe rhand, some BID-* keys in staging do not have TTL, I am not sure if that's correct.

      Some relevant issues:

      https://github.com/breamware/sidekiq-batch/issues/22 (Parameterize BID_EXPIRE_TTL to make it variable)

      https://github.com/breamware/sidekiq-batch/issues/68 (Job never gets set to complete and stats get deleted)

              Unassigned Unassigned
              jgallaso Jose Miguel Gallas Olmedo
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: