Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8122

Lack of documentation for using kafka with SASL

XMLWordPrintable

    • 4
    • False
    • None
    • False
    • 3.0.1.Final

      I wanted to use a kafka cluster with sasl for using debezium 2.7 . After hours of investigation, I've found out that there are few env variables like CONNECT_SASL_MECHANISM,  CONNECT_SECURITY_PROTOCOL and  CONNECT_SASL_JAAS_CONFIG which should be set to connect to kafka. But I've found them in stackoverflow and there was no hint on debezium docs. 

      After I've found the correct values for these variables, I have deployed my postgresql connector but it seemed to be not working. Although debezium wrote messages to it's own internal topics (config, offset, status), there was no messages in my cdc topic. Debezium would create publisher and replication slot on my postgresql instance but seemed to stuck in snapshot process. 

      Sometimes an unrelevent error appeared in my debezium logs ( for example `unable to register MBean`). When I changed log level of debezium to INFO, I saw that each second it printed a message like this with info level:

      ```Cancelled in-flight METADATA request with correlation id 490 due to node -1 being disconnected```

      Long story short. After days of debugging everything, I've read at a random blog post that I should set also the sasl config with producer prefix: CONNECT_PRODUCER_SASL_MECHANISM, CONNECT_PRODUCER_SECURITY_PROTOCOL, CONNECT_PRODUCER_SASL_JAAS_CONFIG.

      It was a very frustrating experience because:

      1. There was no logs to tell me what was the problem
      2. There was nothing in documentation about these values
      3. I can't found these variables even when I grep into debezium source code

              broldan@redhat.com Robert Roldan
              mehdip Mehdi Pourfar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: