Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-13858

Infinispan health check not responding sometimes

    XMLWordPrintable

Details

    • Bug
    • Status: New (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 13.0.6.Final
    • None
    • REST
    • None

    Description

      Currently, we deploy three pods of Infinispan 13.0.6 in one DC and another 3 pods of Infinispan 13.0.6 in anther DC under OpenShift. One of the biggest issue is a lot of time we can see the health check at http://127.0.0.1:11222/rest/v2/cache-managers/default/health/status stop responding and the pods get killed due to failed the health check. We checked all the JVM configurations and the GC was only take about 20-50ms (the cache size less than 200MB). We enabled the debug log of the Infinispan, but cannot really see any obvious problem in the log:

       

      2022-04-26 16:08:27,936 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.statetransfer.StateConsumerImpl] Removing no longer owned entries for cache ___hotRodTopologyCache_hotrod-default
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x63F977FB33D5BD49D503B9D39755F0F6
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xFE5F80F4CE74763B2DEDAA7CE24F27F2
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xAB85D37038BDDCDC607C56EFA28810AB
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x7C7970E75BA3DF681589BE00F49DAA4F
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x52BE86595CC53C803242CF11EA235EBC
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x041352068F6C6160D9CD38CF2E8D41F8
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x8F0F435791D64A7CE4440B9AA593869B
      2022-04-26 16:08:58,583 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x238866F3A5EF2ADED0E433541813E8B9
      2022-04-26 16:08:58,603 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xEDE9E166865BD7E129911A9F84C276E0
      2022-04-26 16:09:23,823 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xC931FFA88895CDAD20CD4F6C3C93A497
      2022-04-26 16:09:23,825 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xC1BA71B5104AC08802435F6BC1243366
      2022-04-26 16:09:27,675 DEBUG (Timer runner-2,infinispan-server-dca-845c7b4cb-njzsr-17472) [org.jgroups.protocols.UNICAST3] infinispan-server-dca-845c7b4cb-njzsr-17472: removing expired connection for infinispan-server-dca-845c7b4cb-knn9g-1190 (240068 ms old) from send_table
      2022-04-26 16:09:27,675 DEBUG (Timer runner-2,infinispan-server-dca-845c7b4cb-njzsr-17472) [org.jgroups.protocols.UNICAST3] infinispan-server-dca-845c7b4cb-njzsr-17472: removing expired connection for infinispan-server-dca-845c7b4cb-knn9g-1190 (240068 ms old) from recv_table
      2022-04-26 16:09:38,028 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xA84F87971CA41A7B571CAEFB4C764937
      2022-04-26 16:09:38,115 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x9996A56AD52F055526ECC41029317606
      2022-04-26 16:09:48,613 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xEBE99F39AAAA22913DE87C39AA0F0FDC
      2022-04-26 16:09:48,613 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xF1D05C4F8709884D7BF5D6C8B5EA8582
      2022-04-26 16:09:48,613 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xEB33B2982C5103C0C5F8432B7F5130F2
      2022-04-26 16:09:48,613 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x92B71C3DC97C6315198FDCB27238B833
      2022-04-26 16:09:48,613 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xAEBDFA9045BBCA626AC3ACD7981E0941
      2022-04-26 16:09:48,613 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xEDE9E166865BD7E129911A9F84C276E0
      2022-04-26 16:09:48,613 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x1449D385C2B9379E66A1B0A14B137049
      2022-04-26 16:09:48,884 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x238866F3A5EF2ADED0E433541813E8B9
      2022-04-26 16:09:48,885 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x8F0F435791D64A7CE4440B9AA593869B
      2022-04-26 16:09:57,993 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xFB4DEFE662E301948C0B1EA5BA4FAD89
      2022-04-26 16:10:13,929 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x496849732A7832711F82CF11D270B94E
      2022-04-26 16:10:38,809 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xF1D05C4F8709884D7BF5D6C8B5EA8582
      2022-04-26 16:10:38,905 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x1449D385C2B9379E66A1B0A14B137049
      2022-04-26 16:11:18,418 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x29815A6BC19D0CFC467E1B1174B73457
      2022-04-26 16:11:29,265 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x238866F3A5EF2ADED0E433541813E8B9
      2022-04-26 16:11:29,278 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x041352068F6C6160D9CD38CF2E8D41F8
      2022-04-26 16:11:38,270 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x22E3ABA27E342442A1205D444329A137
      2022-04-26 16:11:54,126 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0x6503F87FCDA766988BCF722CA64A4182
      2022-04-26 16:12:10,713 DEBUG (Timer runner-2,infinispan-server-dca-845c7b4cb-njzsr-17472) [org.jgroups.protocols.UNICAST3] infinispan-server-dca-845c7b4cb-njzsr-17472: removing expired connection for infinispan-server-dca-845c7b4cb-6r7mv-15796 (240065 ms old) from send_table
      2022-04-26 16:12:10,713 DEBUG (Timer runner-2,infinispan-server-dca-845c7b4cb-njzsr-17472) [org.jgroups.protocols.UNICAST3] infinispan-server-dca-845c7b4cb-njzsr-17472: removing expired connection for infinispan-server-dca-845c7b4cb-6r7mv-15796 (240065 ms old) from recv_table
      2022-04-26 16:12:19,202 DEBUG (non-blocking-thread--p2-t1) [org.infinispan.server.hotrod.ClientListenerRegistry] Channel disconnected, removing event sender listener for id: [B0xF1D05C4F8709884D7BF5D6C8B5EA8582
      2022-04-26 16:12:31,176 INFO  (Thread-1) [org.infinispan.SERVER] ISPN080002: Infinispan Server stopping
      2022-04-26 16:12:31,178 DEBUG (Thread-1) [org.infinispan.rest.RestServer] Stopping server REST-rest-default listening at xxx.xxx.xxx.xxx:11222
      2022-04-26 16:12:31,178 DEBUG (Thread-1) [org.infinispan.server.core.AbstractProtocolServer] Stopping server REST-rest-default listening at xxx.xxx.xxx.xxx:11222
      2022-04-26 16:12:31,178 DEBUG (Thread-1) [org.infinispan.server.core.AbstractProtocolServer] Server REST-rest-default stopped
      2022-04-26 16:12:31,178 DEBUG (Thread-1) [org.infinispan.server.hotrod.HotRodServer] Stopping server HotRod-hotrod-default listening at xxx.xxx.xxx.xxx:11222
      2022-04-26 16:12:31,179 DEBUG (Thread-1) [org.infinispan.registry.impl.InternalCacheRegistryImpl] Unregistering internal cache ___hotRodTopologyCache_hotrod-default
      2022-04-26 16:12:31,179 DEBUG (Thread-1) [org.infinispan.cache.impl.CacheImpl] Stopping cache ___hotRodTopologyCache_hotrod-default on infinispan-server-dca-845c7b4cb-njzsr-17472
      2022-04-26 16:12:31,182 DEBUG (Thread-1) [org.infinispan.topology.LocalTopologyManagerImpl] Node infinispan-server-dca-845c7b4cb-njzsr-17472 leaving cache ___hotRodTopologyCache_hotrod-default

      ***Server process (99) received TERM signal ***

      2022-04-26 16:12:31,193 DEBUG (Thread-1) [org.infinispan.server.core.AbstractProtocolServer] Stopping server HotRod-hotrod-default listening at xxx.xxx.xxx.xxx:11222
      2022-04-26 16:12:31,193 DEBUG (Thread-1) [org.infinispan.server.core.AbstractProtocolServer] Server HotRod-hotrod-default stopped

       

      I'm wondering, except the JVM doing a GC, is there any other possibilities that Infinispan will not respond to health check? Will Infinispan server response health check during the state transferring?

      Attachments

        Activity

          People

            Unassigned Unassigned
            jhua04 Jie Hua
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: