Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-13536

Fix Irac3SitesConflictTest.testConditionalRemove() random failures

    XMLWordPrintable

Details

    Description

      Fix for:

            java.lang.AssertionError: Tombstone map is not empty!
      	at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:237)
      	at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:253)
      	at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:390)
      	at org.infinispan.xsite.AbstractMultipleSitesTest.assertNoDataLeak(AbstractMultipleSitesTest.java:150)
      	at org.infinispan.xsite.irac.Irac3SitesConflictTest.doTest(Irac3SitesConflictTest.java:292)
      	at org.infinispan.xsite.irac.Irac3SitesConflictTest.testConditionalRemove(Irac3SitesConflictTest.java:98)
      
      

      There is a problem with org.infinispan.xsite.AbstractMultipleSitesTest#assertNoDataLeak. First, it should wait for all the IRAC queues are empty before triggering the tombstone cleanup round.
      Otherwise, the cleanup fails.

      Trace logs:

      # LON sends request to NYC to check if the tombstone can be removed
      2021-11-30 18:39:33,957 TRACE (blocking-thread-Irac3SitesConflictTest-NodeA-p48-t1) [org.infinispan.remoting.transport.jgroups.JGroupsTransport] Irac3SitesConflictTest-NodeA sending backup request 16 to SiteMaster(NYC-2): IracCleanupTombstoneCommand{cacheName=defaultcache, key=k0-testConditionalRemove, tombstone=null}
      
      # At the same time, NYC finished replicate the key but it hasn't removed the key from the map yet
      2021-11-30 18:39:33,957 TRACE (jgroups-11,bridge-org.infinispan.xsite.irac.Irac3SitesConflictTest[configMode=OPTIMISTIC_TX_RC],_Irac3SitesConflictTest-NodeD:NYC-2) [org.infinispan.xsite.irac.DefaultIracManager] Replication completed for key 'k0-testConditionalRemove'. /* ... */
      
      # The LON request is processed and it replies with true (== I have the key in the map)
      2021-11-30 18:39:33,957 TRACE (jgroups-5,bridge-org.infinispan.xsite.irac.Irac3SitesConflictTest[configMode=OPTIMISTIC_TX_RC],_Irac3SitesConflictTest-NodeD:NYC-2) [org.infinispan.remoting.transport.jgroups.JGroupsTransport] Irac3SitesConflictTest-NodeD sending response for request 16 to Irac3SitesConflictTest-NodeA:LON-1: SuccessfulResponse(true)
      
      # Finally, the key is removed from the queue (too late, the tombstone will be cleanup in the next round, but the test fails before it happens.)
      2021-11-30 18:39:33,957 TRACE (jgroups-11,bridge-org.infinispan.xsite.irac.Irac3SitesConflictTest[configMode=OPTIMISTIC_TX_RC],_Irac3SitesConflictTest-NodeD:NYC-2) [org.infinispan.xsite.irac.DefaultIracManager] Removing key 'k0-testConditionalRemove'. /* ... */, removed=true
      

      Attachments

        Activity

          People

            pruivo@redhat.com Pedro Ruivo
            pruivo@redhat.com Pedro Ruivo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: