-
Task
-
Resolution: Done
-
Minor
-
0.7.5
-
None
The inclusion of "slot.name" does not work with Postgres and leaves things in a non-obvious state. After creating a new connector with "slot.name" specified:
- curl returns 201 Created, so caller thinks it worked
HTTP/1.1 201 Created
Date: Tue, 19 Jun 2018 20:11:19 GMT
Location: http://localhost:8083/connectors/connect-xyz
Content-Type: application/json
Content-Length: 619
Server: Jetty(9.2.24.v20180105)
- Kafka Connect lists the new connector, which also leads caller to believe it worked
curl http://localhost:8083/connectors
- Kafka Connect logs show error
[2018-06-19 11:01:09,589] INFO 127.0.0.1 - - [19/Jun/2018:17:01:08 +0000] "POST /connectors/ HTTP/1.1" 201 619 1285 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2018-06-19 11:01:11,267] INFO user 'test' connected to database 'test' on PostgreSQL 10.4 on x86_64-apple-darwin17.5.0, compiled by Apple LLVM version 9.1.0 (clang-902.0.39.1), 64-bit with roles:
role 'pg_read_all_settings' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
role 'pg_stat_scan_tables' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
role 'pg_monitor' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
role 'pg_read_all_stats' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
role 'pg_signal_backend' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
role 'test' [superuser: true, replication: false, inherit: true, create role: true, create db: true, can log in: true] (io.debezium.connector.postgresql.PostgresConnectorTask:74)
[2018-06-19 11:01:11,269] INFO Found previous offset source_info[server='localhost', lsn=1A/57E94158, txId=9087, useconds=1529426992355000, snapshot=false] (io.debezium.connector.postgresql.PostgresConnectorTask:88)
[2018-06-19 11:01:11,269] INFO Previous snapshot has completed successfully, streaming logical changes from last known position (io.debezium.connector.postgresql.PostgresConnectorTask:103)
[2018-06-19 11:01:11,271] INFO Requested thread factory for connector PostgresConnector, id = localhost named = records-stream-producer (io.debezium.util.Threads:231)
[2018-06-19 11:01:12,053] DEBUG No replication slot 'xyz-test' is present for plugin 'decoderbufs' and database 'test' (io.debezium.connector.postgresql.connection.PostgresConnection:141)
[2018-06-19 11:01:12,054] DEBUG Creating new replication slot 'xyz-test' for plugin 'DECODERBUFS' (io.debezium.connector.postgresql.connection.PostgresReplicationConnection:95)
[2018-06-19 11:01:12,065] INFO WorkerSourceTask
Unknown macro: {id=connect-xyz-test-0}
Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:328)
[2018-06-19 11:01:12,066] INFO WorkerSourceTaskflushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:345)
[2018-06-19 11:01:12,066] ERROR WorkerSourceTask
Unknown macro: {id=connect-xyz-test-0}
Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)
io.debezium.jdbc.JdbcConnectionException: ERROR: syntax error
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initReplicationSlot(PostgresReplicationConnection.java:136)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.<init>(PostgresReplicationConnection.java:79)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.<init>(PostgresReplicationConnection.java:38)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$ReplicationConnectionBuilder.build(PostgresReplicationConnection.java:349)
at io.debezium.connector.postgresql.PostgresTaskContext.createReplicationConnection(PostgresTaskContext.java:63)
at io.debezium.connector.postgresql.RecordsStreamProducer.<init>(RecordsStreamProducer.java:75)
at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:105)
at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:45)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:164)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:260)
at org.postgresql.replication.fluent.logical.LogicalCreateSlotBuilder.make(LogicalCreateSlotBuilder.java:48)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initReplicationSlot(PostgresReplicationConnection.java:102)
... 15 more
[2018-06-19 11:01:12,068] ERROR WorkerSourceTaskTask is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:173)
- Postgres logs show error: "ERROR: syntax error"
2018-06-19 11:01:12.053 MDT [17813] LOG: execute <unnamed>: select * from pg_replication_slots where slot_name = $1 and database = $2 and plugin = $3
2018-06-19 11:01:12.053 MDT [17813] DETAIL: parameters: $1 = 'xyz-test', $2 = 'test', $3 = 'decoderbufs'
2018-06-19 11:01:12.054 MDT [17813] LOG: disconnection: session time: 0:00:00.776 user=test database=test host=127.0.0.1 port=56011
2018-06-19 11:01:12.057 MDT [17814] LOG: connection received: host=127.0.0.1 port=56012
2018-06-19 11:01:12.058 MDT [17814] LOG: replication connection authorized: user=test
2018-06-19 11:01:12.063 MDT [17814] ERROR: syntax error
- Postgres shows no rows in pg_replication_slots
# select * FROM pg_replication_slots;
slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn
----------+---------------------------------------------------------------------------------------------------------------
(0 rows)
once in this state, the following gets it working again:
- remove the existing connector (curl -X DELETE http://localhost:8083/connectors/connect-xyz-test)
- remove "slot.name" from the configuration
- retry same "curl -i -X POST ...." as above
- everything works, replication slot is created
versions in use:
- Debezium 0.7.5
- Postgres 10.4
- Kafka 2.11
- Mac OS X 10.13.5
- relates to
-
DBZ-699 PostgresReplicationConnection doesn't close jdbc connection
-
- Closed
-