When the debezium server tries to send a new change event to kinesis sink it is sending a very long partition key, bigger than 256 bytes, what raises and error and kill the service. It occurs only on tables with multiple primary keys.

 

ERROR:

'partitionKey' failed to satisfy constraint: Member must have length less than or equal to 256

 

RECORD KEY:

{
    "schema": {
        "type": "struct",
        "fields": [
           

{                 "type": "int32",                 "optional": false,                 "default": 0,                 "field": "project_id"             }

,
           

{                 "type": "int32",                 "optional": false,                 "default": 0,                 "field": "country_id"             }

,
           

{                 "type": "int32",                 "optional": false,                 "default": 0,                 "field": "quotagroup"             }

,
           

{                 "type": "int32",                 "optional": false,                 "default": 0,                 "field": "row_index"             }

        ],
        "optional": false,
        "name": "dbz.ows_production.project_quotagroup_completes.Key"
    },
    "payload":

{         "project_id": 6,         "country_id": 7,         "quotagroup": 0,         "row_index": 1     }

}

 

When trying to solve this, I found a blog post from Gunnar (https://debezium.io/blog/2018/08/30/streaming-mysql-data-changes-into-kinesis/) that mention exactly what is happening:

"Now one could think of mapping the key from Debezium’s change data events to the Kinesis partition key, but partition keys are limited to a length of 256 bytes. Depending on the length of primary key column(s) in the captured tables, this might not be enough. So a safer option is to create a hash value from the change message key and use that as the partition key."

 

Saying that I think that the best option is to follow Gunnar idea and implement a partition key based on hashs instead of the record key itself.