Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6813

Documentation Request - Property File Configuration - Off-Heap Event Buffering with Embedded Infinispan

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Major Major
    • 2.4.0.Beta1
    • None
    • documentation
    • None

      What Debezium connector do you use and what version?

      Oracle - 2.3.1 Final

      What is the connector configuration?

      name=enquesta-cdc-connector-1
      connector.class=io.debezium.connector.oracle.OracleConnector
      tasks.max=1
      database.server.name=sdsenq
      database.hostname=redacted
              database.port=1521
              database.user=redacted
              database.password=redacted
      database.dbname=sdsenq
      table.include.list=DBZUSER.DEBEZIUM_SIGNAL,ENQUESTACIS.CI911DISTRICT,ENQUESTACIS.CIACCOUNTACTIVECODE,ENQUESTACIS.CIACCOUNTMETERSTATUS,ENQUESTACIS.CIACCOUNTSTATUS,ENQUESTACIS.CIAGE,ENQUESTACIS.CIAMITYPE,ENQUESTACIS.CIAPPLICATION,ENQUESTACIS.CIAPPTAXDISTRICT,ENQUESTACIS.CIASSETTAXDISTRICT,ENQUESTACIS.CIBALANCEBREAKDOWN,ENQUESTACIS.CIBILLINGCYCLE,ENQUESTACIS.CICLASSTYPE,ENQUESTACIS.CICONSTRUCTIONTYPE,ENQUESTACIS.CICOUNTYCODE,ENQUESTACIS.CIDEVICECONFIG,ENQUESTACIS.CIDEVICELOCATION,ENQUESTACIS.CIDEVICEMODEL,ENQUESTACIS.CIDEVICEPARAMETER,ENQUESTACIS.CIDEVICESIZE,ENQUESTACIS.CIDEVICETYPE,ENQUESTACIS.CIDISCOUNTCODE,ENQUESTACIS.CIEMAIL,ENQUESTACIS.CIITEMCODE,ENQUESTACIS.CIMASTERREPORTCODE,ENQUESTACIS.CINAMEPREFIX,ENQUESTACIS.CINEIGHBORHOODCODE,ENQUESTACIS.CIPHONE,ENQUESTACIS.CIPOLICEDISTRICT,ENQUESTACIS.CIPROPERTYCLASS,ENQUESTACIS.CIREPORTCODEDESC,ENQUESTACIS.CISECONDARYPARTY,ENQUESTACIS.CISERVICETYPE,ENQUESTACIS.CISTATE,ENQUESTACIS.CISUBDIVISION,ENQUESTACIS.CITAXCODE,ENQUESTACIS.CITAXTYPE,ENQUESTACIS.CITOWN,CIS.FCIADDR,CIS.FCIAPREM,CIS.FCIBAL,CIS.FCICIS,CIS.FCICMTR,CIS.FCIDEVICE,CIS.FCIINSTAL,CIS.FCIITEM,CIS.FCIPREM,CIS.FCIREGISTER,CIS.FCISERVI,CIS.FDGCMAS,CIS.FRFCMAS,CIS.FRTCMAS,CIS.FSWCMAS,CIS.FTXCMAS,CIS.FUTCMAS,CIS.FWACMAS
      database.history.kafka.bootstrap.servers=redacted:9092
      database.history.kafka.topic=schema-changes-cdc.inventory
      schema.history.internal.kafka.topic=schema-history-cdc
      schema.history.internal.kafka.bootstrap.servers=redacted:9092
      topic.prefix=sds_
      database.history.store.only.captured.tables.ddl=true
      internal.log.mining.query.logs.for.snapshot.offset=true
      signal.data.collection=SDSENQ.DBZUSER.DEBEZIUM_SIGNAL
      value.converter.schemas.enable=false
      key.converter.schemas.enable=false
      log.mining.archive.destination.name=log_archive_dest_2
      log.mining.strategy=online_catalog
      log.mining.query.filter.mode=in
      log.mining.buffer.type=infinispan_embedded
      log.mining.buffer.infinispan.cache.transactions=<local-cache name="transactions"><persistence passivation="false"><file-store read-only="false" preload="true" shared="false"><data path="/data02/cdc/cache/data"/><index path="/data02/cdc/cache/index"/></file-store></persistence></local-cache>

      What is the captured database version and mode of depoyment?

       Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
      Version 19.20.0.0.0 

      Oracle Cloud Infrastructure VM

      What behaviour do you expect?

      After experiencing an OutOfMemory exception after a large transaction and researching fixes, I found ccranfor@redhat.com 's article on Debezium Oracle Connector performance where the option of using off-heap event buffering is described.  I would like to utilize the Embedded Infinispan option, but although the Infinispan XML settings for log.mining.buffer.infinispan.cache.transactions are described in the documentation (and seem to parse ok in the config properties file), examples of the XML configuration values for the other required properties

      • log.mining.buffer.infinispan.cache.events,
      • log.mining.buffer.infinispan.cache.processed_transactions,
      • log.mining.buffer.infinispan.cache.schema_changes

      are not provided and cause an exception on startup.

       

      java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 3 error(s):
      The 'log.mining.buffer.infinispan.cache.events' value is invalid: A value is required
      The 'log.mining.buffer.infinispan.cache.processed_transactions' value is invalid: A value is required
      The 'log.mining.buffer.infinispan.cache.schema_changes' value is invalid: A value is required

       

      What behaviour do you see?

      The exception above

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes

      Do you have the connector logs, ideally from start till finish?

      I can provide them if necessary

      How to reproduce the issue using our tutorial deployment?

      Replicate the provided configuration in a test environment

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      Improve/Expand the documentation describing off-heap event buffering 

       

            ccranfor@redhat.com Chris Cranford
            chris.f.whelan@gmail.com Chris Whelan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: