Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9521

Events are mistakenly skipped during commit dispatch when LOB is enabled

XMLWordPrintable

    • Critical

      Recently, we noticed that we were experiencing data loss after restarting the connector. We tried upgrading to version 3.2.3. Final.
      However, the data loss increased even further, even with the connector active (without restarting). The logs showed numerous messages like "[io.deb.con.ora.log.buf.BufferedLogMinerStreamingChangeEventSource] (debezium-oracleconnector-change-event-source-coordinator) Skipping event 5 in transaction 3f000c00a2c91c00 - has already been sent."

      LogFile is attached

      logging settings
      quarkus.log.level=INFO
      quarkus.log.category."io.debezium.connector.oracle".level=ALL
      quarkus.log.category."io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource".level=INFO

      We have lost almost 2000 records at Table15 during 1 minute capruerd in the attached log file.

      Our logminer settings
      debezium.source.lob.enabled=true
      debezium.source.max.batch.size=60000
      debezium.source.max.queue.size=120000
      debezium.source.log.mining.batch.size.max=300000
      debezium.source.log.mining.batch.size.default=100000
      debezium.source.log.mining.batch.size.min=50000
      debezium.source.log.mining.batch.size.increment=50000
      debezium.source.query.fetch.size=60000
      debezium.source.log.mining.transaction.retention.ms=2400000

      We have reselect postprocessor turned on.

        1. b4000000469c1200.masked.csv
          1.15 MB
        2. dlog04.log_v.gz
          8.27 MB
        3. dlog4.log.gz
          2.41 MB

              ccranfor@redhat.com Chris Cranford
              nickolay.k Nikolay Kozlov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: