Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5983

Custom topic creation group cleanup.policy is not working

    XMLWordPrintable

Details

    • Bug
    • Resolution: Can't Do
    • Major
    • None
    • None
    • mongodb-connector
    • None
    • False
    • None
    • False
    • Moderate

    Description

      Bug report

      What Debezium connector do you use and what version?

      MongoDB source connector 1.8.1 and version 2.0.0 Final

      What is the connector configuration?

       

      topic.creation.dev2.include: "dev-api\\.a,dev-api\\.b,dev-api\\.c"
      topic.creation.dev2.replication.factor: 3
      topic.creation.dev2.partitions: 5
      topic.creation.dev2.cleanup.policy: "delete"
      topic.creation.dev2.compression.type: "lz4"
      topic.creation.dev2.retention.ms: 864000000 # 10 days in ms
      
      topic.creation.heartbeat.include: "__mongodb_heartbeats\\.debezium"
      topic.creation.heartbeat.replication.factor: 3
      topic.creation.heartbeat.partitions: 1
      topic.creation.heartbeat.cleanup.policy: "delete"
      topic.creation.heartbeat.compression.type: "lz4"
      
      # Registers the custom topic creation groups
      topic.creation.groups: "dev2,heartbeat"  

      What is the captured database version and mode of deployment?

      MongoDB Alas 4.2.23 Replicaset

      What behavior do you expect?

      The topic cleanup.policy to be delete after new topics are created

      What behavior do you see?

      The topic cleanup.policy is compact

      Do you see the same behavior using the latest released Debezium version?

      Yes, Just tested it with 2.1.1

      Do you have the connector logs, ideally from start to finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      ```

      Created topic '(name=debezium.dev-api.a, numPartitions=5, replicationFactor=3, replicasAssignments=null, configs={compression.type=lz4, cleanup.policy=compact})' using creation group TopicCreationGroup{name='default', inclusionPattern=.*, exclusionPattern=, numPartitions=5, replicationFactor=3, otherConfigs={compression.type=lz4, cleanup.policy=compact}} (org.apache.kafka.connect.runtime.AbstractWorkerSourceTask) [task-thread-strimzi-kafka-connect-mongodb-debezium-source-connector-0]

      ```

      How to reproduce the issue using our tutorial deployment?

      Add the configuration above to the connector configs for a nonexisting topic. 

      Set the `topic.creation.enable` to true on connect cluster and run the connector.

      Attachments

        Activity

          People

            Unassigned Unassigned
            thr.firoozian@gmail.com Tahereh Firoozian (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: