-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
RHDG Operator CSV 8.3.6 GA
-
None
-
False
-
None
-
False
-
-
-
-
-
After the server is killed by OOM internal 'org.infinispan.LOCKS' cache enters DEGRADED_MODE in which it stays even after the servers restart.
Node 0 after restart
$ oc logs test-pod-degradation-after-oom-0 | grep LOCKS 10:42:21,945 DEBUG (main) [org.infinispan.registry.impl.InternalCacheRegistryImpl] Registering internal cache org.infinispan.LOCKS [EXCLUSIVE] 10:42:22,586 DEBUG (main) [org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl] Starting statically defined caches: [___event_log_cache, ___protobuf_metadata, org.infinispan.CONFIG, org.infinispan.COUNTER, ___script_cache, org.infinispan.CLIENT_SERVER_TX_TABLE, org.infinispan.LOCKS] 10:42:22,963 DEBUG (main) [org.infinispan.manager.DefaultCacheManager] Creating cache org.infinispan.LOCKS on test-pod-degradation-after-oom-0-12841 10:42:22,974 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.LocalTopologyManagerImpl] Node test-pod-degradation-after-oom-0-12841 joining cache org.infinispan.LOCKS 10:42:22,976 DEBUG (jgroups-6,test-pod-degradation-after-oom-0-12841) [org.infinispan.topology.LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.LOCKS: CacheTopology{id=7, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[test-pod-degradation-after-oom-1-22716], persistentUUIDs=[13a20f7d-8ec6-4871-a64c-4b24c9c5f286]} 10:42:22,976 DEBUG (jgroups-6,test-pod-degradation-after-oom-0-12841) [org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl] Updating availability for cache org.infinispan.LOCKS: AVAILABLE -> DEGRADED_MODE 10:42:22,980 DEBUG (main) [org.infinispan.cache.impl.CacheImpl] Started cache org.infinispan.LOCKS on test-pod-degradation-after-oom-0-12841 10:42:24,561 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.ClusterCacheStatus] Recovered 1 partition(s) for cache org.infinispan.LOCKS: [CacheTopology{id=7, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[test-pod-degradation-after-oom-1-22716], persistentUUIDs=[13a20f7d-8ec6-4871-a64c-4b24c9c5f286]}] 10:42:24,561 ERROR (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN000319: After merge (or coordinator change), cache still hasn't recovered all its data and must stay in degraded mode. Current members are [], lost members are [4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716], stable members are [4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716] 10:42:24,561 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] ISPN000521: Cache org.infinispan.LOCKS recovered after merge with topology = CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[], persistentUUIDs=[]}, availability mode DEGRADED_MODE 10:42:24,561 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100007: After merge (or coordinator change), recovered members [4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716] with topology id 8 10:42:24,562 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.ClusterCacheStatus] ISPN000519: Updating stable topology for cache org.infinispan.LOCKS, topology CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[4da404e3-96ad-44f3-a766-a514e3b02aa3, test-pod-degradation-after-oom-1-22716], persistentUUIDs=[a3424f2d-954c-4945-9386-84d972c88edd, 13a20f7d-8ec6-4871-a64c-4b24c9c5f286]} 10:42:24,562 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] After merge, cache org.infinispan.LOCKS is staying in degraded mode 10:42:24,564 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.topology.LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.LOCKS: CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, test-pod-degradation-after-oom-1-22716: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[], persistentUUIDs=[]} 10:42:24,565 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] Cache org.infinispan.LOCKS is not available, updating the actual members only 10:42:28,360 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] Cache org.infinispan.LOCKS is not available, updating the actual members only 10:42:28,795 DEBUG (jgroups-17,test-pod-degradation-after-oom-0-12841) [org.infinispan.partitionhandling.impl.PreferConsistencyStrategy] Cache org.infinispan.LOCKS not available (DEGRADED_MODE), postponing rebalance for joiner test-pod-degradation-after-oom-1-61014
Node 1 after restart
$ oc logs test-pod-degradation-after-oom-1 | grep LOCKS 10:42:27,745 DEBUG (main) [org.infinispan.registry.impl.InternalCacheRegistryImpl] Registering internal cache org.infinispan.LOCKS [EXCLUSIVE] 10:42:28,430 DEBUG (main) [org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl] Starting statically defined caches: [___event_log_cache, ___protobuf_metadata, org.infinispan.CONFIG, org.infinispan.COUNTER, ___script_cache, org.infinispan.CLIENT_SERVER_TX_TABLE, org.infinispan.LOCKS] 10:42:28,780 DEBUG (main) [org.infinispan.manager.DefaultCacheManager] Creating cache org.infinispan.LOCKS on test-pod-degradation-after-oom-1-61014 10:42:28,794 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.topology.LocalTopologyManagerImpl] Node test-pod-degradation-after-oom-1-61014 joining cache org.infinispan.LOCKS 10:42:28,797 DEBUG (jgroups-6,test-pod-degradation-after-oom-1-61014) [org.infinispan.topology.LocalTopologyManagerImpl] Updating local topology for cache org.infinispan.LOCKS: CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[4da404e3-96ad-44f3-a766-a514e3b02aa3: 125+131, daa532eb-fcd9-4ba4-ac2f-350ae17e3042: 131+125]}, pendingCH=null, unionCH=null, actualMembers=[], persistentUUIDs=[]} 10:42:28,797 DEBUG (jgroups-6,test-pod-degradation-after-oom-1-61014) [org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl] Updating availability for cache org.infinispan.LOCKS: AVAILABLE -> DEGRADED_MODE 10:42:28,801 DEBUG (main) [org.infinispan.cache.impl.CacheImpl] Started cache org.infinispan.LOCKS on test-pod-degradation-after-oom-1-61014
- relates to
-
JDG-3991 OpenShift: Server killed by OOM restarts forever after
- Closed