Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9717

Schema history restricted by 'snapshot.include.collection.list'

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.4.0.CR1
    • 3.4.0.Beta1
    • oracle-connector
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Important

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      debezium-connector-oracle-3.0.0

      What is the connector configuration?

      {
      "connector.class": "io.debezium.connector.oracle.OracleConnector",
      "database.hostname": "oracle",
      "database.port": "1521",
      "database.user": "c##dbzuser",
      "database.password": "dbz",
      "database.dbname": "FREE",
      "decimal.handling.mode": "double",
      "topic.prefix": "CHECKSIGN",
      "topic.creation.default.partitions": "5",
      "topic.creation.default.replication.factor": "1",
      "key.converter": "org.apache.kafka.connect.storage.StringConverter",
      "value.converter": "io.confluent.connect.avro.AvroConverter",
      "value.converter.schema.registry.url": "http://schemaregistry:8081",
      "table.include.list": "BASEADM.STTM_ACCOUNT_CLASS",
      "snapshot.max.threads": "1",
      "snapshot.locking.mode": "none",
      "tombstones.on.delete": "false",
      "schema.history.internal.kafka.bootstrap.servers": "broker:29092",
      "schema.history.internal.kafka.topic": "schema-changes.checksign",
      "snapshot.mode": "initial",
      "log.mining.strategy": "redo_log_catalog",
      "custom.metric.tags": "table=partner",
      "transforms": "Reroute",
      "transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
      "transforms.Reroute.topic.regex": "CHECKSIGN.BASEADM.(.*)",
      "transforms.Reroute.topic.replacement": "PUBLIC.STREAMING.FLEX.$1.DEV",
      "signal.enabled.channels": "source,kafka",
      "signal.data.collection": "FREE.BASEADM_1.DEBEZIUM_SIGNAL",
      "signal.kafka.topic": "CHECKSIGN.SIGNALS",
      "signal.kafka.bootstrap.servers": "broker:29092",
      "schema.history.internal.store.only.captured.tables.ddl":"true",
      "snapshot.include.collection.list": "FREE.BASEADM.STTM_ACCOUNT_CLASS",
      "incremental.snapshot.chunk.size":"100000"
      }

      What is the captured database version and mode of deployment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      Tested it on-prem in a dockerized environment. My Oracle image is: 
      container-registry.oracle.com/database/free:23.9.0.0

      What behavior do you expect?

      snapshot.include.collection.list should not restrict the schema of the capturable tables or the schema of signal.data.collection.

      What behavior do you see?

      if schema.history.internal.store.only.captured.tables.ddl:true, and either table.include.list contains a table with a schema what does not appear in snapshot.include.collection.list for any table or the same for signal.data.collection's schema, then an exception is thrown on connector startup:

      io.debezium.DebeziumException: Unable to find relational table model for 'FREE.BASEADM_1.DEBEZIUM_SIGNAL', there may be an issue with your include/exclude list configuration.

      Do you see the same behaviour using the latest released Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      Yes, tested it with 

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      Yes, log file attached

      How to reproduce the issue using our tutorial deployment?

      1. set "schema.history.internal.store.only.captured.tables.ddl":"true"
      2. use 2 different schemas in "table.include.list": "BASEADM.STTM_ACCOUNT_CLASS, BASEADM_1.PARTNER"
      3. only add table(s) from one schema to "snapshot.include.collection.list": "FREE.BASEADM.STTM_ACCOUNT_CLASS"

      or

      1. set "schema.history.internal.store.only.captured.tables.ddl":"true"
      2. let "signal.data.collection" be in a schema, what doesn't appear in "snapshot.include.collection.list"
         

        "signal.data.collection": "FREE.BASEADM_1.DEBEZIUM_SIGNAL""snapshot.include.collection.list": "FREE.BASEADM.STTM_ACCOUNT_CLASS"

      Here is a detailed analysis and the root causes of the issue, what i've discussed with ccranfor@redhat.com :
      https://groups.google.com/g/debezium/c/sBqpVyj3j-g

       

              Unassigned Unassigned
              solidsofthungary@gmail.com Gergely Jahn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: