Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5638

Postgres connector Mbean error on multi-AZ failover

    XMLWordPrintable

Details

    • False
    • None
    • False

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      The bug

      When I pause a connector, cause a multi-AZ failover which is promoting the hot standby in another region to be the master, and then resume the connector I get an error about an mbean instance already existing.

      What Debezium connector do you use and what version?

      debezium/debezium-connector-postgresql:1.9.2

      What is the connector configuration?

      ```
      "name" = "test_db_outbox_connector"
      "connector.class" = "io.debezium.connector.postgresql.PostgresConnector"
      "errors.log.enable" = "true"
      "errors.log.include.messages" = "true"
      "snapshot.mode" = "initial"
      "plugin.name" = "pgoutput"
      "tasks.max" = "1"
      "database.hostname" = local.test_db.host
      "database.port" = local.test_db.port
      "database.user" = local.test_db.username
      "database.dbname" = local.test_db.db_name
      "database.server.name" = "test_db_outbox"
      "slot.name" = "outbox_connector_slot"
      "key.converter" = "org.apache.kafka.connect.storage.StringConverter"
      "key.converter.schemas.enable" = "false"
      "value.converter" = "org.apache.kafka.connect.json.JsonConverter"
      "value.converter.schema.registry.url" = local.schema_registry_url
      "value.converter.schemas.enable" = "false"
      "value.converter.auto.register.schemas" = "false"
      "include.schema.changes" = "false"
      "table.include.list" = "public.outbox"
      "publication.autocreate.mode" = "disabled"
      "publication.name" = "outbox_publication"
      "poll.interval.ms" = "500"

      "task.shutdown.graceful.timeout.ms" = "15000"
      "status.update.interval.ms" = "5000"

      "offset.flush.timeout.ms" = "30000"

      "buffer.memory" = "268435456"
      "offset.flush.interval.ms" = "15000"
      "session.timeout.ms" = "30000"
      "heartbeat.interval.ms" = "3000"
      "heartbeat.topics.prefix" = "debezium-heartbeat"
      "transforms" = "outbox"
      "transforms.outbox.type" = "io.debezium.transforms.outbox.EventRouter"
      "transforms.outbox.table.fields.additional.placement" = "ordering_group_id:header:OrderingGroupId,event_name:header:EventName,generated_at:header:GeneratedAt,event_id:header:EventId,origin:header:Origin"
      "transforms.outbox.table.field.event.id" = "event_id"
      "transforms.outbox.table.field.event.key" = "ordering_group_id"
      "transforms.outbox.table.field.event.payload" = "payload"
      "transforms.outbox.table.expand.json.payload" = "true"
      "transforms.outbox.route.by.field" = "event_name"
      "transforms.outbox.route.topic.replacement" = "test_db.public.outbox"
      ```

      What is the captured database version and mode of deployment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      RDS Postgres v12.10

      What behaviour do you expect?

      I expect the connector to restart

      What behaviour do you see?

      The connector fails to restart and logs an error around mbean instance already existing.

      Do you see the same behaviour using the latest relesead Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      Have not tested this

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      ```

      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:116) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.RuntimeException: Unable to register the MBean 'debezium.postgres:type=connector-metrics,context=snapshot,server=test_db_outbox' at io.debezium.metrics.Metrics.register(Metrics.java:77) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:101) ... 5 more Caused by: javax.management.InstanceAlreadyExistsException: debezium.postgres:type=connector-metrics,context=snapshot,server=test_db_outbox at java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436) at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855) at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955) at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890) at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320) at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at io.debezium.metrics.Metrics.register(Metrics.java:73) ... 6 more    
      ```

      How to reproduce the issue using our tutorial deployment?

      Assuming this was the tutorial than no:

      https://debezium.io/documentation/reference/1.9/tutorial.html

      This was for postgres following a muli-AZ failover

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      Similar to these issues:

      https://issues.apache.org/jira/browse/KAFKA-3992

      https://issues.redhat.com/browse/DBZ-5138

      https://issues.redhat.com/browse/DBZ-4792

       

      I am also willing to just disable mbean but I have yet to find a way to do that even when following these docs:

      https://debezium.io/documentation/reference/stable/operations/monitoring.html#kafka-connect-jmx-environment-variables

      I have tried all of these so far:

      ```

      • name: "CONNECT_JMXLOG4J"
        value: "true"
      • name: "CONNECT_JMX_OPTS"
        value: "-Dzookeeper.jmx.log4j.disable=true"
      • name: "KAFKA_JMX_OPTS"
        value: "-Dzookeeper.jmx.log4j.disable=true"
      • name: "CONNECT_KAFKA_JMX_OPTS"
        value: "-Dzookeeper.jmx.log4j.disable=true"
        ```
         

      Attachments

        Activity

          People

            Unassigned Unassigned
            ekeric13 Eric Kennedy (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: