Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4064

Infinispan SPI throws NPE with more than one connector configured to the same Oracle database

    XMLWordPrintable

Details

    Description

      We are using Oracle 19c with LOB disabled using the infinispan SPI. This runs fine with a single connector, but once we have more than one connector connecting to the same Oracle instance (to reduce the amount of tables handled by a single connector, thus by a single KafkaConnect instance), these random NPEs are appearing. Restarting the connector then resumes the work as normal up until it crashes again. This can happen like 20 times per day.

      We realize that running with more than one connector to the same database is not a recommended practice, this also made us decide to revert back to a single connector. This removes this exception altogether.

      It starts with this error:

      2021-09-22 10:24:47    java.lang.NullPointerException: Null keys are not supported!
      2021-09-22 10:24:47        at java.base/java.util.Objects.requireNonNull(Objects.java:246)
      2021-09-22 10:24:47        at org.infinispan.cache.impl.CacheImpl.assertKeyNotNull(CacheImpl.java:202)
      2021-09-22 10:24:47        at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:541)
      2021-09-22 10:24:47        at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:526)
      2021-09-22 10:24:47        at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:522)
      2021-09-22 10:24:47        at org.infinispan.cache.impl.AbstractDelegatingCache.containsKey(AbstractDelegatingCache.java:384)
      2021-09-22 10:24:47        at org.infinispan.cache.impl.EncoderCache.containsKey(EncoderCache.java:601)
      2021-09-22 10:24:47        at io.debezium.connector.oracle.logminer.processor.infinispan.InfinispanLogMinerEventProcessor.processRow(InfinispanLogMinerEventProcessor.java:206)
      2021-09-22 10:24:47        at io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor.processResults(AbstractLogMinerEventProcessor.java:150)
      2021-09-22 10:24:47        at io.debezium.connector.oracle.logminer.processor.infinispan.InfinispanLogMinerEventProcessor.process(InfinispanLogMinerEventProcessor.java:176)
      2021-09-22 10:24:47        at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:144)
      2021-09-22 10:24:47        at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:54)
      2021-09-22 10:24:47        at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:165)
      2021-09-22 10:24:47        at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:127)
      2021-09-22 10:24:47        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      2021-09-22 10:24:47        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      2021-09-22 10:24:47        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      2021-09-22 10:24:47        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      2021-09-22 10:24:47        at java.base/java.lang.Thread.run(Thread.java:834)
      

      And 20 minutes or so later we get this exception:

      java.lang.NullPointerException
          at io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor.processRow(AbstractLogMinerEventProcessor.java:162)
          at io.debezium.connector.oracle.logminer.processor.infinispan.InfinispanLogMinerEventProcessor.processRow(InfinispanLogMinerEventProcessor.java:210)
          at io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor.processResults(AbstractLogMinerEventProcessor.java:150)
          at io.debezium.connector.oracle.logminer.processor.infinispan.InfinispanLogMinerEventProcessor.process(InfinispanLogMinerEventProcessor.java:176)
          at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:144)
          at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:54)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:165)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:127)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:834)
      

      Our connector config is as follows:

      {
          "config": {
              "connector.class": "io.debezium.connector.oracle.OracleConnector",
              "snapshot.mode": "initial",
              "snapshot.locking.mode": "none",
              "tasks.max": "1",
              "database.server.name": "myserver",
              "database.url": "myjdbcstring",
              "database.dbname": "my_svc",
              "event.processing.failure.handling.mode": "warn",
              "topics": "mytopic",
              "log.mining.sleep.time.min.ms": "1000",
              "log.mining.sleep.time.max.ms": "10000",
              "log.mining.batch.size.min": "10000",
              "log.mining.batch.size.max": "200000",
              "log.mining.view.fetch.size": "20000",
              "log.mining.sleep.time.default.ms": "1000",
              "log.mining.sleep.time.increment.ms": "1000",
              "log.mining.archive.log.hours": "5",
              "log.mining.archive.log.only.mode": "false",
              "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_2",
              "log.mining.strategy": "online_catalog",
              "decimal.handling.mode": "string",
              "log.mining.buffer.type": "infinispan",
              "log.mining.buffer.location": "/data/infinispan/myconnector",
              "heartbeat.interval.ms": "60000",
              "provide.transaction.metadata": "true",
              "tombstones.on.delete": "false",
              "poll.interval.ms": "60000",
              "database.history.kafka.bootstrap.servers": "my-kafka-bootstrap-host.fqdn:9094",
              "database.history.kafka.topic": "myserver.history",
              "database.history.store.only.captured.tables.ddl": "true",
              "database.history.skip.unparseable.ddl": "true",
              "table.include.list": "MYSCHEMA.MYTABLE1,MYSCHEMA.MYTABLE2"
          }
      }
      

      Attachments

        Issue Links

          Activity

            People

              ccranfor@redhat.com Chris Cranford
              jorn.argelo@mn.nl Jorn Argelo
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: