Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-5460

'org.infinispan.LOCKS' cache stays in DEGREDED_MODE after OOM on OpenShift

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • RHDG Operator CSV 8.3.6 GA
    • Openshift Images
    • None

    Description

      After the server is killed by OOM internal 'org.infinispan.LOCKS' cache enters DEGRADED_MODE in which it stays even after the servers restart.

      Node 0 after restart

      $ oc logs test-pod-degradation-after-oom-0 | grep LOCKS
      10:42:21,945 DEBUG (main) [org.infinispan.registry.impl.InternalCacheRegistryImpl] Registering internal cache org.infinispan.LOCKS [EXCLUSIVE]
      10:42:22,586 DEBUG (main) [org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl] Starting statically defined caches: [___event_log_cache, ___protobuf_metadata, org.infinispan.CONFIG, org.infinispan.COUNTER, ___script_cache, org.infinispan.CLIENT_SERVER_TX_TABLE, org.infinispan.LOCKS]
      10:42:22,963 DEBUG (main) [org.infinispan.manager.DefaultCacheManager] Creating cache org.infinispan.LOCKS on test-pod-degradation-after-oom-0-12841
      10:42:22,974 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.LocalTopologyManagerImpl] Node test-pod-degradation-after-oom-0-12841 joining cache org.infinispan.LOCKS
      10:42:22,976 DEBUG (jgroups-6,test-pod-degradation-after-oom-0-12841) [org.infinispan.topology.LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.LOCKS: CacheTopology{id=7, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[test-pod-degradation-after-oom-1-22716], persistentUUIDs=[13a20f7d-8ec6-4871-a64c-4b24c9c5f286]}
      10:42:22,976 DEBUG (jgroups-6,test-pod-degradation-after-oom-0-12841) [org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl] Updating availability for cache org.infinispan.LOCKS: AVAILABLE -> DEGRADED_MODE
      10:42:22,980 DEBUG (main) [org.infinispan.cache.impl.CacheImpl] Started cache org.infinispan.LOCKS on test-pod-degradation-after-oom-0-12841
      10:42:24,561 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.ClusterCacheStatus] Recovered 1 partition(s) for cache org.infinispan.LOCKS: [CacheTopology{id=7, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[test-pod-degradation-after-oom-1-22716], persistentUUIDs=[13a20f7d-8ec6-4871-a64c-4b24c9c5f286]}]
      10:42:24,561 ERROR (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN000319: After merge (or coordinator change), cache still hasn't recovered all its data and must stay in degraded mode. Current members are [], lost members are [4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716], stable members are [4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716]
      10:42:24,561 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] ISPN000521: Cache org.infinispan.LOCKS recovered after merge with topology = CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[], persistentUUIDs=[]}, availability mode DEGRADED_MODE
      10:42:24,561 INFO  (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100007: After merge (or coordinator change), recovered members [4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716] with topology id 8
      10:42:24,562 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.ClusterCacheStatus] ISPN000519: Updating stable topology for cache org.infinispan.LOCKS, topology CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716], persistentUUIDs=[a3424f2d-954c-4945-9386-84d972c88edd, 13a20f7d-8ec6-4871-a64c-4b24c9c5f286]}
      10:42:24,562 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] After merge, cache org.infinispan.LOCKS is staying in degraded mode
      10:42:24,564 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.topology.LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.LOCKS: CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[], persistentUUIDs=[]}
      10:42:24,565 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] Cache org.infinispan.LOCKS is not available, updating the actual members only
      10:42:28,360 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] Cache org.infinispan.LOCKS is not available, updating the actual members only
      10:42:28,795 DEBUG (jgroups-17,test-pod-degradation-after-oom-0-12841) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] Cache org.infinispan.LOCKS not available (DEGRADED_MODE), postponing rebalance for joiner test-pod-degradation-after-oom-1-61014
      

      Node 1 after restart

      $ oc logs test-pod-degradation-after-oom-1 | grep LOCKS
      10:42:27,745 DEBUG (main) [org.infinispan.registry.impl.InternalCacheRegistryImpl] Registering internal cache org.infinispan.LOCKS [EXCLUSIVE]
      10:42:28,430 DEBUG (main) [org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl] Starting statically defined caches: [___event_log_cache, ___protobuf_metadata, org.infinispan.CONFIG, org.infinispan.COUNTER, ___script_cache, org.infinispan.CLIENT_SERVER_TX_TABLE, org.infinispan.LOCKS]
      10:42:28,780 DEBUG (main) [org.infinispan.manager.DefaultCacheManager] Creating cache org.infinispan.LOCKS on test-pod-degradation-after-oom-1-61014
      10:42:28,794 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.LocalTopologyManagerImpl] Node test-pod-degradation-after-oom-1-61014 joining cache org.infinispan.LOCKS
      10:42:28,797 DEBUG (jgroups-6,test-pod-degradation-after-oom-1-61014) [org.infinispan.topology.LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.LOCKS: CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, daa532eb-fcd9-4ba4-ac2f-350ae17e3042: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[], persistentUUIDs=[]}
      10:42:28,797 DEBUG (jgroups-6,test-pod-degradation-after-oom-1-61014) [org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl] Updating availability for cache org.infinispan.LOCKS: AVAILABLE -> DEGRADED_MODE
      10:42:28,801 DEBUG (main) [org.infinispan.cache.impl.CacheImpl] Started cache org.infinispan.LOCKS on test-pod-degradation-after-oom-1-61014
      

      Attachments

        Issue Links

          Activity

            People

              remerson@redhat.com Ryan Emerson
              pdrobek@redhat.com Pavel Drobek
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: