Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6963

Missing events from Oracle 19c

    XMLWordPrintable

Details

    • Critical

    Description

      Bug report

      What Debezium connector do you use and what version?

      Oracle Connector 2.3.2.Final

      What is the connector configuration?

      We have in total 7 connectors running on the same db instance. This is the config of one of them:

      {     
      "connector.class": "io.debezium.connector.oracle.OracleConnector",     
      "key.converter": "io.confluent.connect.avro.AvroConverter",     
      "key.converter.schema.registry.url": "http://schemaurl:8081",     
      "value.converter": "io.confluent.connect.avro.AvroConverter",     
      "value.converter.schema.registry.url": "http://schemaurl:8081",     
      "schema.name.adjustment.mode": "avro",     
      "database.url": "jdbc:oracle:thin:@//db-url",     
      "database.user": "DSL_RO",     
      "database.password": "ENC[AES256_GCM,data:SJUTzVObXsgLjXjp,iv:aXSv0DMxR8niTOkOksGxqvQ9tMph6aeX6FcW9Rac/Qw=,tag:pwmPnKYLatC2KIInCwT0vA==,type:str]",     
      "database.dbname": "db-name",     
      "topic.prefix": "ASE1",     
      "table.include.list": "ASEDBUSR.ABRDVRS,ASEDBUSR.ABRDVRSUNT,ASEDBUSR.ACTIVITY,ASEDBUSR.ADRESSE,ASEDBUSR.ANTRFRAGENANTW,ASEDBUSR.ARBUN,ASEDBUSR.BANK,ASEDBUSR.BEZIEHUNG,ASEDBUSR.BILANZAUSGLEICH,ASEDBUSR.BUCHSTOFFV,ASEDBUSR.BUCHUNGSTOFF",     
      "snapshot.select.statement.overrides": "ASEDBUSR.ABRDVRS,ASEDBUSR.ABRDVRSUNT,ASEDBUSR.ACTIVITY,ASEDBUSR.ADRESSE,ASEDBUSR.ANTRFRAGENANTW,ASEDBUSR.ARBUN,ASEDBUSR.BANK,ASEDBUSR.BEZIEHUNG,ASEDBUSR.BILANZAUSGLEICH,ASEDBUSR.BUCHSTOFFV,ASEDBUSR.BUCHUNGSTOFF",     
      "snapshot.select.statement.overrides.ASEDBUSR.ABRDVRS": "SELECT * FROM ASEDBUSR.ABRDVRS ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.ABRDVRSUNT": "SELECT * FROM ASEDBUSR.ABRDVRSUNT ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.ACTIVITY": "SELECT * FROM ASEDBUSR.ACTIVITY ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.ADRESSE": "SELECT * FROM ASEDBUSR.ADRESSE ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.ANTRFRAGENANTW": "SELECT * FROM ASEDBUSR.ANTRFRAGENANTW ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.ARBUN": "SELECT * FROM ASEDBUSR.ARBUN ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.BANK": "SELECT * FROM ASEDBUSR.BANK ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.BEZIEHUNG": "SELECT * FROM ASEDBUSR.BEZIEHUNG ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.BILANZAUSGLEICH": "SELECT * FROM ASEDBUSR.BILANZAUSGLEICH ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.BUCHSTOFFV": "SELECT * FROM ASEDBUSR.BUCHSTOFFV ORDER BY CREATED ASC, PKEY ASC",     "snapshot.select.statement.overrides.ASEDBUSR.BUCHUNGSTOFF": "SELECT * FROM ASEDBUSR.BUCHUNGSTOFF ORDER BY CREATED ASC, PKEY ASC",     
      "snapshot.fetch.size": "10000",     
      "snapshot.max.threads": 3,     
      "event.processing.failure.handling.mode": "fail",     
      "tombstones.on.delete": "true",     
      "message.key.columns": "ASEDBUSR.ABRDVRS:BOID;ASEDBUSR.ABRDVRSUNT:BOID;ASEDBUSR.ACTIVITY:BOID;ASEDBUSR.ADRESSE:BOID;ASEDBUSR.ANTRFRAGENANTW:BOID;ASEDBUSR.ARBUN:BOID;ASEDBUSR.BANK:BOID;ASEDBUSR.BEZIEHUNG:BOID;ASEDBUSR.BILANZAUSGLEICH:BOID;ASEDBUSR.BUCHSTOFFV:BOID;ASEDBUSR.BUCHUNGSTOFF:BOID",     
      "snapshot.mode": "initial",     
      "log.mining.strategy": "online_catalog",     
      "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_10",     
      "lob.enabled": "true",     
      "unavailable.value.placeholder": "previous_value_unchanged",     
      "log.mining.transaction.retention.hours": "24",     
      "internal.log.mining.log.query.max.retries": "10",     
      "time.precision.mode": "connect",     
      "tasks.max": "1",     
      "schema.history.internal.store.only.captured.tables.ddl": "true",     
      "schema.history.internal.kafka.bootstrap.servers": "bootstrap:9092",     
      "schema.history.internal.kafka.topic": "ase.schema-changes.asedbusr",     
      "schema.history.internal.consumer.security.protocol": "SASL_SSL",     
      "schema.history.internal.producer.security.protocol": "SASL_SSL",     
      "schema.history.internal.consumer.sasl.mechanism": "SCRAM-SHA-512",     
      "schema.history.internal.producer.sasl.mechanism": "SCRAM-SHA-512",     
      "schema.history.internal.consumer.sasl.jaas.config": "ENC[AES256_GCM,data:uABy6fODTAzj97XYO5Z4cmh6DclvFeC6NtrTanaD80BptIhImIj501uTsuDvh/fc96ZZNaM9Q6lbszL1OjQ36iH7mhBEQaUrRoVKV1c5oB0FBqkXKiu+h5MIqynrQhaVGK2i36GTW4WADW3qmXrqueeIWg==,iv:w+B6U1/MgpgFOTZIKeDHJwzAjZ1z/ps4Hsjo4WOurSE=,tag:8EtLJXncKir26U4zKEk13Q==,type:str]",     "schema.history.internal.producer.sasl.jaas.config": "ENC[AES256_GCM,data:5QjqvYPFBi6BJawSWtDyT/dfBy7AvLs2vyrLFDhKolHBUqtvH2qEe7aSUB/DIg5y0sOQOM9ymiLplo3tJXMIyhVCv9oNYh5AYBFUFfYiS1Wq3O0DDJ3FueTaw5F9YJE5ZMayIfb/dELkoqY+MFctSryA1Q==,iv:tvIx//LpBc/aAKc91uevNaDLRcS9eRL0mmVgx/Mi57k=,tag:z9oIHek4cP35hQzaCDZFKA==,type:str]",     
      "notification.enabled.channels": "sink",     
      "notification.sink.topic.name": "ASE1.SNAPSHOT.STATUS",     
      "transforms": "changes,moveHeadersToValue,unwrap",    
       "transforms.changes.type": "io.debezium.transforms.ExtractChangedRecordState",     
      "transforms.changes.header.changed.name": "Changed",     
      "transforms.moveHeadersToValue.type": "io.debezium.transforms.HeaderToValue",     
      "transforms.moveHeadersToValue.headers": "Changed",     
      "transforms.moveHeadersToValue.fields": "changes",     
      "transforms.moveHeadersToValue.operation": "move",     
      "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",     
      "transforms.unwrap.add.fields": "source.table:META_SRC_TABLENAME,source.ts_ms:META_SRC_TS,op:META_SRC_OP,source.scn:META_SRC_SCN,source.snapshot:META_SRC_SNAPSHOT,source.user_name:META_SRC_USER,changes:META_SRC_CHANGES",     
      "transforms.unwrap.delete.handling.mode": "rewrite",     
      "transforms.unwrap.drop.tombstones": "false",     
      "sops": {         
      "kms": null,         
      "gcp_kms": null,         
      "azure_kv": null,         
      "hc_vault": null,         
      "age": null,         
      "lastmodified": "2022-11-15T16:38:33Z",         
      "mac": "ENC[AES256_GCM,data:wzrF6qR6CXHWFuDWgv3DEQ0cD+jMCcX2ekmEgiPA4nRx2NxU1ICtB6JAeS7tqOkeirHDpGD3uWguk8Eoyu0mYNLXXCEZH4c43XL1G+uArWw6N7HXDvFPFRKCsc9lZdixTeERe5FLCrYy1PKMgoDZMJBlYlCdVz8wv28Dur9DJvs=,iv:rqzMz9tQpBaV7KACMkuiUsAm6EkmQCzn5aR/GNe3hz4=,tag:TRUBXv8SxD21BrcMERjAQw==,type:str]",     
      "pgp": [             
      {                 "created_at": "2022-11-15T16:38:33Z",                 
      "enc": "-----BEGIN PGP MESSAGE-----\n\nhQEMA243CXYlIAr3AQf/aZ+OoTLb79bfctbWI+FzEdA9yerMSGhoTcwHJbEHI+VS\nSMOj1Q0e2peGcuxtahtG00jJqowEhhA++QELmTCbkYeWRqNLu9atksiixBTw/8hM\nUwQcTAYBPDuhROTYgPIhTDvFlk6axIKYmYMCGx2rNsm1bDfLDd5Qh0muoU4iPBN0\nxLAVb+kycoYC0sMFcPgA9zdmn1RzkyGCfrG5I9kN1dBYrIgJBHGv1BNPA4CO/AyD\nHVlW7sD8twrik89pf572ovdIDVVWDpAfM1yVuE7eGUUwKWeseB82wLcsO7y2OrbI\nP/5OI7lNz0Os5Kby48n7lOpOSAvjOozdAWH0C+pX+NJcAdgzVIEjd0TO0AzFxbZl\nteccD0sE8vGIVzjw8lXdF8JY3lFm2L7iJNKAzjQblanWBbcVHE9AGqUJHe/vKVZo\nZby6fjxVu3DYYwQW2TVkG/BXulEm8xMLSC9anpk=\n=0an4\n-----END PGP MESSAGE-----\n",                 
      "fp": "F7067DB106402AABEBEF13F8C6E64E19DC477D85"             }         ],         
      "encrypted_regex": "(database.password|schema.history.internal.consumer.sasl.jaas.config|schema.history.internal.producer.sasl.jaas.config)",   
      "version": "3.7.1"     
      } 
      }

      What is the captured database version and mode of depoyment?

      Oracle 19c on-Prem, non CDB, no RAC

      What behaviour do you see?

      Some events get lost resp. aren't emitted. In the attached file there is an example of missed events in one table. We have more of these missed events at different times and also in other tables. We parallelly load the data from the same database with Informatica Powerexchange and then do a comparison between the data delivered by Informatica and by Debezium. In the attached file you see the events we get from Informatica and also the statements from the Logminer content.

      Do you have the connector logs, ideally from start till finish?

      No warn or error entries (but running Kafka connect on INFO level)

       

      Attachments

        Issue Links

          Activity

            People

              ccranfor@redhat.com Chris Cranford
              renerh René Rütter
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: