Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4834

Adding new table with incremental snapshots not working

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Hide

      I used the debezium/connect:1.9 and the oracle/database:19.3.0-ee docker image.

      0. Initialize oracle database

      docker-compose exec oracle mkdir /opt/oracle/oradata/recovery_area
cat setup-oracle-logminer.sql | docker-compose exec -T oracle sqlplus /nolog
cat test-data.sql | docker-compose exec -T oracle sqlplus "myuser/SECRET@//localhost:1521/MYPDB"

      1. Install connector with only the signaling table in the "table.include.list".

      curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d@connector1.json

      2. Delete connector

      curl -i -X DELETE -H "Accept:application/json" localhost:8083/connectors/ora_con

      3. Add the same connector again with an extra table in the "table.include.list".

      curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d@connector2.json

      4. Trigger incremental snapshot

      INSERT INTO C##LOGMINER.DEBEZIUM_SIGNAL (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["MYPDB.MYUSER.ITEMS"],"type":"incremental"}');
      Show
      I used the debezium/connect:1.9 and the oracle/database:19.3.0-ee docker image. 0. Initialize oracle database docker-compose exec oracle mkdir /opt/oracle/oradata/recovery_area cat setup-oracle-logminer.sql | docker-compose exec -T oracle sqlplus /nolog cat test-data.sql | docker-compose exec -T oracle sqlplus "myuser/SECRET@ //localhost:1521/MYPDB" 1. Install connector with only the signaling table in the "table.include.list" . curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d@connector1.json 2. Delete connector curl -i -X DELETE -H "Accept:application/json" localhost:8083/connectors/ora_con 3. Add the same connector again with an extra table in the "table.include.list". curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d@connector2.json 4. Trigger incremental snapshot INSERT INTO C##LOGMINER.DEBEZIUM_SIGNAL (id, type, data) VALUES( 'ad-hoc-1' , 'execute-snapshot' , '{ "data-collections" : [ "MYPDB.MYUSER.ITEMS" ], "type" : "incremental" }' );

    Description

      When adding a new table to an existing oracle connector and then triggering an incremental snapshot of that new table, the snapshot is not actually executed and a warning appears

      WARN   Oracle|ora_con|streaming  Schema not found for table 'MYPDB.MYUSER.ITEMS', known tables [MYPDB.C##LOGMINER.DEBEZIUM_SIGNAL]   [io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource]

      The connector seems to only know the tables from the initial configuration.

      I expect a new topic to be created (the table contains data), but there is none.

      I've attached the trace log and the connector configurations.

      {{}}

      Attachments

        1. connector2.json
          0.8 kB
        2. connector1-1.json
          0.8 kB
        3. incremental-1.log
          14.69 MB
        4. test-data.sql
          0.6 kB
        5. setup-oracle-logminer.sql
          3 kB
        6. incremental-2.log
          12.72 MB

        Activity

          People

            ccranfor@redhat.com Chris Cranford
            hendrik.luedemann Hendrik Lüdemann (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: