XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: 2.7.0.Beta1
Affects Version/s: 2.4.1.Final, 2.5.0.Beta1
Component/s: sqlserver-connector
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

Bug report

For bug reports, provide this information, please:

What Debezium connector do you use and what version?

SQL-Server connector 2.4.1 (https://repo1.maven.org/maven2/io/debezium/debezium-connector-sqlserver/2.4.1.Final/debezium-connector-sqlserver-2.4.1.Final-plugin.tar.gz)

What is the connector configuration?

KafkaConnect:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.6.0
  replicas: 1
  bootstrapServers: debezium-cluster-kafka-bootstrap:9092
  logging:
    type: inline
    loggers:
      connect.root.logger.level: DEBUG
        #connect.root.logger.level: DEBUG
  config:
    producer.max.request.size: 5000000
    max.request.size: 5000000
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  build:
    output:
      type: docker
      image: XXXXXXXXXX.dkr.ecr.eu-central-1.amazonaws.com/debezium-connect-sqlserver:latest
      pushSecret: debezium-ecr-secret
    plugins:
      - name: debezium-sqlserver-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-sqlserver/2.4.1.Final/debezium-connector-sqlserver-2.4.1.Final-plugin.tar.gz
  template:
    pod:
      imagePullSecrets:
        - name: awsecr-cred

KafkaConnector:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: debezium-connector-sqlserver
  namespace: kafka
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.sqlserver.SqlServerConnector
  tasksMax: 1
  config:
    log4j.rootLogger: DEBUG, stdout
    tasks.max: 1
    database.hostname: 192.168.3.152
    database.port: 1433
    database.user: "sa"
    database.password: "Passw0rd"
    database.names: "debeziumdb,debeziumdb2"
    database.server.name: dec2000
    database.encrypt: false
    table.include.list: "dbo.debezium_signal,dbo.randomtable,dbo.randomtable2"
    signal.data.collection: "debeziumdb.dbo.debezium_signal"
    schema.history.internal.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092
    schema.history.internal.kafka.topic: dbhistory.dec2000
    decimal.handling.mode: string
    topic.prefix: dec2000

What is the captured database version and mode of depoyment?

SQL Server 2016 Enterprise on Windows as well as SQL Server 2022 on Linux via Kubernetes.

Here is a T-SQL Script to create my test environment:

use master
GO
 --alter database  debeziumdb set single_user with rollback immediate
 --alter database  debeziumdb2 set single_user with rollback immediate
DROP DATABASE IF EXISTS debeziumdb;
DROP DATABASE IF EXISTS debeziumdb2;
GO


GO
CREATE DATABASE debeziumdb2;
CREATE DATABASE debeziumdb;
GO
USE debeziumdb;
GO
DROP TABLE IF EXISTS randomtable;


CREATE TABLE randomtable (
    ID INT IDENTITY(1,1) PRIMARY KEY,
    randomtext NVARCHAR(500)
);
GO
DROP TABLE IF EXISTS debezium_signal
CREATE TABLE [dbo].[debezium_signal](
	[id] [varchar](42) NOT NULL,
	[type] [varchar](32) NOT NULL,
	[data] [varchar](2048) NULL,
PRIMARY KEY CLUSTERED 
(
	[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]


GO
DECLARE @Counter INT = 1;


WHILE @Counter <= 50
BEGIN
    DECLARE @RandomText NVARCHAR(500);
    
    SET @RandomText = (
        SELECT TOP 1
            LEFT(CONVERT(NVARCHAR(MAX), NEWID()), 500)
        FROM sys.all_objects
    );
    
    INSERT INTO randomtable (randomtext) VALUES (@RandomText);
    
    SET @Counter = @Counter + 1;
END;
GO
EXEC sys.sp_cdc_enable_db
GO
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name   = N'randomtable', 
@role_name     = NULL, 
@supports_net_changes = 0
GO
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name   = N'debezium_signal', 
@role_name     = NULL, 
@supports_net_changes = 0


GO


USE debeziumdb2; 
GO
DROP TABLE IF EXISTS randomtable2;


CREATE TABLE randomtable2 (
    ID INT IDENTITY(1,1) PRIMARY KEY,
    randomtext NVARCHAR(500)
);

GO

DECLARE @Counter INT = 1;
WHILE @Counter <= 50
BEGIN
    DECLARE @RandomText NVARCHAR(500);
    SET @RandomText = (
        SELECT TOP 1
            LEFT(CONVERT(NVARCHAR(MAX), NEWID()), 500)
        FROM sys.all_objects
    );
    INSERT INTO randomtable2 (randomtext) VALUES (@RandomText);
    SET @Counter = @Counter + 1;
END;
GO
GO
EXEC sys.sp_cdc_enable_db
GO
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name   = N'randomtable2', 
@role_name     = NULL, 
@supports_net_changes = 0


GO

What behaviour do you expect?

The connector should register multiple signals, including "execute-snapshot":

2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'schema-changes' using class 'io.debezium.pipeline.signal.actions.SchemaChanges' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'snapshot-window-open' using class 'io.debezium.pipeline.signal.actions.snapshotting.OpenIncrementalSnapshotWindow' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'execute-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'snapshot-window-close' using class 'io.debezium.pipeline.signal.actions.snapshotting.CloseIncrementalSnapshotWindow' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'log' using class 'io.debezium.pipeline.signal.actions.Log' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'stop-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.StopSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'resume-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.ResumeIncrementalSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'pause-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.PauseIncrementalSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]

The connector should create a snapshot after creating a signal:

INSERT INTO debezium_signal (id, type, data) VALUES (NEWID(), 'execute-snapshot', '{"data-collections": ["debeziumdb.dbo.randomtable"]}')

What behaviour do you see?

The log output above mentioning the registration of multiple signals is not produced. Also, after creating the "execute-snapshot" signal, the following log is produced:

2023-12-06 13:21:13,165 WARN [debezium-connector-sqlserver|task-0] Signal '0B91F382-4AE5-40EF-BF7E-94A1EDF9B54B' has been received but the type 'execute-snapshot' is not recognized (io.debezium.pipeline.signal.SignalProcessor)

However, if only one database is configured, the behavior is as expected:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: debezium-connector-sqlserver
  namespace: kafka
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.sqlserver.SqlServerConnector
  tasksMax: 1
  config:
    log4j.rootLogger: DEBUG, stdout
    tasks.max: 1
    database.hostname: 192.168.3.152
    database.port: 1433
    database.user: "sa"
    database.password: "Passw0rd"
    database.names: "debeziumdb"
    database.server.name: dec2000
    database.encrypt: false
    table.include.list: "dbo.debezium_signal,dbo.randomtable"
    signal.data.collection: "debeziumdb.dbo.debezium_signal"
    schema.history.internal.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092
    schema.history.internal.kafka.topic: dbhistory.dec2000
    decimal.handling.mode: string
    topic.prefix: dec2000

Do you see the same behaviour using the latest relesead Debezium version?

Yes, as well with the current 2.5.0 Beta Version.

Do you have the connector logs, ideally from start till finish?

Here are the logs with two databases configured and no signals being registered: https://gist.github.com/guyincognito-io/d28e9b47045f5aaaec87b8fbf6da908b

Here are logs with one database configured and all signals being registered, as expected:

https://gist.github.com/guyincognito-io/a60c4d458639c58ac7587aef3f0a0901

How to reproduce the issue using our tutorial deployment?

I am not sure, I can look into this if required.

is duplicated by

DBZ-7583 SQL Server signal actions not recognized with more than one database

Closed

Assignee:: Mario Fiore Vitale

Reporter:: Florian Hedtstück (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2023/12/11 9:50 AM

Updated:: 2024/05/10 8:02 AM

Details

Description

Bug report

What Debezium connector do you use and what version?

What is the connector configuration?

What is the captured database version and mode of depoyment?

What behaviour do you expect?

What behaviour do you see?

Do you see the same behaviour using the latest relesead Debezium version?

Do you have the connector logs, ideally from start till finish?

How to reproduce the issue using our tutorial deployment?

Attachments

Issue Links

Activity

People

Dates

Hide