Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-14238

Memcached server: corrupted response

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • 14.0.2.Final, 15.0.0.Dev01
    • 14.0.1.Final
    • None
    • None

    Description

      While doing a stress test with the new changes to the memcached decoder, the spymemcached throws IllegalStateException: No read operation, meaning the server is sending something that isn't supposed to send.

      Adding trace/debug messages, found the following pattern. This is the last request done by this decoder instance (spymemcached closes the connection after the exception)

      2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Resetting parameters using instance @21234f16
2022-10-18 17:02:57,536 DEBUG (jgroups-27,pedro-desktop-25592) [org.infinispan.server.memcached.MemcachedDecoder] Sending response 1752 for instance 21234f16, rsp=[B@637f1839, throw=null
2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Decode using instance @21234f16
2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Decode header using instance @21234f16
2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Operation: 'get'
2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Decode using instance @21234f16
2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Decoding key using instance @21234f16
2022-10-18 17:02:57,536 TRACE (jgroups-27,pedro-desktop-25592) [org.infinispan.server.memcached.MemcachedDecoder] Write response using instance @21234f16: [B0x53544F5245440D0A
2022-10-18 17:02:57,536 DEBUG (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Queueing response 1753 for instance 21234f16, operation=GetRequest
2022-10-18 17:02:57,536 DEBUG (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Sending response 1753 for instance 21234f16, rsp=UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 1030, cap: 1030), throw=null
2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Write response using instance @21234f16: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 1030, cap: 1030)
2022-10-18 17:02:57,536 TRACE (non-blocking-thread--p2-t18) [org.infinispan.server.memcached.MemcachedDecoder] Resetting parameters using instance @21234f16

      The test shows a concurrent set() and get() request. The set() ack is delivered by JGroups while the get() is being handled.

      The theory is that Netty queues the writeAndFlush invocation (the JGroups threads is not a Netty thread) but the get() request is response is sent immediately.

      Attachments

        Issue Links

          is caused by

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) ISPN-14207 Make MemcachedDecoder non blocking

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              pruivo@redhat.com Pedro Ruivo
              pruivo@redhat.com Pedro Ruivo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: