Uploaded image for project: 'Keycloak'
  1. Keycloak
  2. KEYCLOAK-18518

Expired cache objects in infinispan cache are never garbage collected and lead to out of memory

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Done
    • 13.0.1, 14.0.0
    • 15.0.0
    • Infinispan
    • None
    • Hide

      Checkout https://github.com/ahahu/keycloak_cache_expiry_oom
      run "docker compose up -d"
      and watch the tests output on the console.

      You can connect to the processes via jconsole or Visual VM to watch the heap space:
      service:jmx:http-remoting-jmx://localhost:9991
      service:jmx:http-remoting-jmx://localhost:9992
      credentials: jmxuser, password

      Keycloak Console credentials: admin / Pa55w0rd

      Show
      Checkout https://github.com/ahahu/keycloak_cache_expiry_oom run "docker compose up -d" and watch the tests output on the console. You can connect to the processes via jconsole or Visual VM to watch the heap space: service:jmx:http-remoting-jmx://localhost:9991 service:jmx:http-remoting-jmx://localhost:9992 credentials: jmxuser, password Keycloak Console credentials: admin / Pa55w0rd
    • Workaround Exists
    • Hide

      Add <expiration lifespan="2000000000000"/> configuration to every distributed and replicated cache.

      Alternatively, apply the fix-cache-expiration-1.cli script.

      Show
      Add <expiration lifespan="2000000000000"/>  configuration to every distributed and replicated cache. Alternatively, apply the  fix-cache-expiration-1.cli script.
    • NEW
    • NEW

    Description

      In the past we already had issues with event releated entites in the work cache that were never garbage collected (see KEYCLOAK-11683). Since release 13.0.0 the issue got much worse since also session related objects are no longer garbage collected. This is probably because since KEYCLOAK-16755 the session expiration relies on inifinispan cache expiration. And while the actual cache expiration does work (at least that's what the statistics show), the cached items are not garbage collected, so that sooner or later a out of memory occurs.

      We have setup a docker compose based project for reproduction (see "steps to reproduce"), that just does direct grant based authentication requests in a loop. The SSO Session timeout is set to 10 minutes. We could reproduce the issue in standalone and standalone HA mode.
      This is probably not limited to "direct grants", but this might facilitate the problem, because neither a logout nor fetching of already timed out sessions happens.
      While (voluntarily) limiting the heap to 256M the OOM happens after about 90.000 requests.

      Here are the results for a test run. The used heap memory goes up, until it hits the limit:

      Meanwhile the cache statistics show, that the expiration is working fine:

       

      These objects don't get garbage collected, even after a forced GC:

      A heap dump analysis shows that those objects are still referenced by infinispan DefaultSegementedDataContainer: 

       

       

       

      Attachments

        1. cache-stats-3-after-long-pause.png
          cache-stats-3-after-long-pause.png
          146 kB
        2. configure-caches.cli
          2 kB
        3. fix-cache-expiration.cli
          1 kB
        4. fix-cache-expiration-1.cli
          1 kB
        5. image-2021-06-23-13-47-12-021.png
          image-2021-06-23-13-47-12-021.png
          336 kB
        6. image-2021-06-23-13-51-50-292.png
          image-2021-06-23-13-51-50-292.png
          342 kB
        7. image-2021-06-24-21-03-50-502.png
          image-2021-06-24-21-03-50-502.png
          253 kB
        8. keycloak_oom_cache_stats.png
          keycloak_oom_cache_stats.png
          83 kB
        9. keycloak_oom_heap.png
          keycloak_oom_heap.png
          51 kB
        10. keycloak_oom_objects_on_heap.png
          keycloak_oom_objects_on_heap.png
          22 kB
        11. keycloak_reference_path-1.png
          keycloak_reference_path-1.png
          93 kB
        12. node-1-keycloak-14-heap.png
          node-1-keycloak-14-heap.png
          122 kB

        Issue Links

          Activity

            People

              hmlnarik@redhat.com Hynek Mlnařík
              ahdpag Achim Hügen (Inactive)
              Votes:
              6 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: