Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3540

Multiple kafka listeners for a single broker not working

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • 1.5.0.Final, 1.6.0.Alpha1
    • container-images
    • None
    • False
    • False
    • I am very sorry there was a typo in the `KAFKA_LISTENERS` env var.
    • Hide

      On a macOS 11.2 (BIg Sur) and Linux/GNU Debian 10, it throws the mentioned error, for docker version 20.10.X.

      version: '3.3'

services: 
  zookeeper: 
    container_name: debezium/zookeeper:latest
    expose: 
      - "2181"
    environment: 
      - ZOOKEEPER_CLIENT_PORT=2181
  kafka: 
    image: debezium/kafka:latest # debezium/kafka:1.5
    container_name: kafka
    expose: 
      - "9092"
    ports: 
      - "9095:9095"
    environment: 
      - KAFKA_HEAP_OPTS=-Xmx512M -Xms512M
      - KAFKA_ADVERTISED_LISTENERS=INT://kafka:9092,EXT://localhost:9095
      - KAFKA_LISTENER=EXT://localhost:9095,INT://kafka:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXT:PLAINTEXT,INT:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INT
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_HOST_NAME=kafka
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - ZOOKEEPER_CONNECT= zookeeper:2181

      Zookeeper image is also debezium/zookeeper:latest with a vanila configuration.

      Show
      On a macOS 11.2 (BIg Sur) and Linux/GNU Debian 10, it throws the mentioned error, for docker version 20.10.X. version: ' 3 . 3 ' services: zookeeper: container_ name: debezium/zookeeper:latest expose: - " 2181 " environment: - ZOOKEEPER_CLIENT_PORT= 2181 kafka: image: debezium/kafka:latest # debezium/kafka: 1 . 5 container_ name: kafka expose: - " 9092 " ports: - " 9095 : 9095 " environment: - KAFKA_HEAP_OPTS=-Xmx 512 M -Xms 512 M - KAFKA_ADVERTISED_LISTENERS=INT://kafka: 9092 ,EXT://localhost: 9095 - KAFKA_LISTENER=EXT://localhost: 9095 ,INT://kafka: 9092 - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXT:PLAINTEXT,INT:PLAINTEXT - KAFKA_INTER_BROKER_LISTENER_NAME=INT - KAFKA_ZOOKEEPER_CONNECT=zookeeper: 2181 - KAFKA_ADVERTISED_HOST_NAME=kafka - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR= 1 - ZOOKEEPER_CONNECT= zookeeper: 2181 Zookeeper image is also debezium/zookeeper:latest with a vanila configuration.

      I am currently using debezium/kafka docker image inside my docker network and I am trying to connect to kafka container with an external kafka client. Moreover there are other containers inside my network such as kafdrop or debezium/connect-db that are also communicating with the kafka broker. Therefore I need two kafka listeners one to bind for requests internal of the docker network and another to connect externally.

      The environment session of my kafka docker compose looks like this:

          environment: 
      - KAFKA_ADVERTISED_LISTENERS=INT://kafka:9092,EXT://localhost:9095
      - KAFKA_LISTENER=EXT://localhost:9095,INT://kafka:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXT:PLAINTEXT,INT:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INT

      However, I get the following error when docker-compose-ing up in a debezium/kafka:1.5

      kafka    | Using KAFKA_LISTENERS=PLAINTEXT://{kafka-dockerip}:9092 and KAFKA_ADVERTISED_LISTENERS=INT://kafka:9092,EXT://localhost:9095
kafka    | 2021-05-19 13:10:23,161 - INFO  [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean
kafka    | 2021-05-19 13:10:23,847 - INFO  [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka    | 2021-05-19 13:10:23,866 - ERROR [main:MarkerIgnoringBase@159] - Exiting Kafka due to fatal exception
kafka    | java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://{kafka-dockerip}:9092': No security protocol defined for listener PLAINTEXT

      If I run the same docker-compose file in the debezium/kafka:latest my external advertised listener is ignored and I get the same error:

      kafka    | Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://{kafka-dockerip}:9092
kafka    | 2021-05-19 13:09:17,995 - INFO  [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean
kafka    | 2021-05-19 13:09:18,694 - ERROR [main:MarkerIgnoringBase@159] - Exiting Kafka due to fatal exception
kafka    | java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://{kafka-dockerip}:9092': No security protocol defined for listener PLAINTEXT
kafka    |      at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:289)

      If I run the same docker container on confluentinc/cp-kafka:latest it works flawlessly. I guess the problem is the time (or when) the security protocol takes place.

            Unassigned Unassigned
            dcguim@gmail.com Daniel Guimaraes (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: