Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-842

"no known snapshots" error when DBs rows are large

XMLWordPrintable

      On Debezium 0.8.1.Final, with postgres 9.6 / wal2json commit d2b7fef021c46e0d429f2c1768de361069e58696, we have encountered connector crashes when processing large db rows.

      For example, consider the following table:

      CREATE TABLE test_table (
    id integer PRIMARY KEY,
    x integer,
    data jsonb
);

      Suppose we insert some db rows containing large blocks of json data:

      INSERT INTO test_table (id, x, data) VALUES
    (1, 1, '{"key": "really long value UPpLXMEESxlYEdtVVTBvfrkyvIQmVzXGiPnOasKCxWRIZjCvlgpsXMwhCEjfkSGkOFvtzNuTgELcqndZtwPUKkzYBGxWkvvxTSPBADprPhHmoYYXQWEIGkCtwDxbBYFatblIBbtQHFwUNAWXazzOkkpVxdTfeBgLeWTYUMbrIofOdUxmvXuoAZHsSEXQbCvoCUkYrNQIypfWoHlmfIaAaQeXuqxOhUWzzyGEZjxHMqiECOtIMLJoxMuSkrFlOXdZmiXXrUKPCVmaYMJNhNpNeSVxIkrxrIgTvTLtwIHmDpBjqKcgPFbfCiVQKQaGKMhcVGJkUMYwStsysZnmMYGbtWCxrfzTqsVGVooPUcMfODkODpgGBkTErgBFRedYGQTIANYGpqJnJqALhbFtOPUuSecWXCdBbKTaPTYsXLcfpFtJYSRdYRmgrcPeyiXVWosRdgHnRJdKdZUOthOLEDcZwRUcSkzVgmZIRCleLFkVvyZZPLjUpyWzldiYhIZFSkRRnzLjCrnNGDLzhoQoYOVWnBjjtNJMaylIVAdriAppJCGUQTblabqyTkRFotyBQFNusmQevROsMUybfpmLfokBwJMTFguwQQfqDFpEdMsQdHooJoiQRYggBlPkAkIooGLLKrUZmkJxwXCsVtkOsjirOkPHzGuaFmKufjKwYtmuBxpvmZroUsakLTyLLgLlFoeWFunyDgrZqQJYJIMFZTFkwsuBfzFInECbylsNlIEnzPIpizslyMWyIlikLQotIBVlNWEUKHXDGQNAOIBonotAtlyQuxyGxidIAhJCkexbfyxrcCoKfBvCQTvAmoHABumkuBAIqAuQMEdKLuuqkBMNkuYCvbshyENYBOmwsTaCkmAVhKWMonNeqSTgnECvJGysLgKZjXdFWysgPEhtIiFsOwSHWHJCMXgzNkmZfIlQTWRBlLWCnebDPYZdjJavMwAenDomZlRdeQVwfHfGdQrLrAAfpsEyoHWeGrgYzUXkwySJFgJyBEfVfbUGJJcHzlvvihzEtDjElisqaWKVTIsmtdATUCYfyzjDICqgNjzgwyYMUwgmmvPGoduAJSuInRGYXAPcpiRaAhLwxBWDjVhoGLVHRFYNPRDmNYoxAoMFbReeIvNVkDmCNSCJkCRfzuvrsVIcPSmiydwPDyCYKPmXIIoJCxtsGEqahKMwzTqNIjvyoKIEDQhftYYuPvsNuscBpZUllaeEraeYdLAlxEQbGSUyLbFNpdCNLkauXcfNadqFCBglRZTLlGFIrxaNXYEWebpahPhvdRkBNRUvkbYtYTcsYMtCfUaOqauoiTdUvYWHuDlIzBzJUEGdZVPgrwfhsBkCsJsEmiBYHWtvGwHgkDbFuauwPCOwybxggAIvCzMVEFhkHRxmXuUsqYFKjCmCdugKJuELMWjKVnYoVPIGGJCSyfzcHpSxVLKRYEyLBmEfZjLZfwOXaBLPcqItkrGLlTYrXOkpQrqRmMzdZJTQNarAvfUqeUWoiwwgeNhavbPxmaUQSwxFxkFHcPzHYrZtCURabLxlDMuJalHgXmFxWaakScrimOGImiDIDyWRazlXksqoPiPJXWQsvxZXFvbegDoFpWToBbQbYcjIQUIlJqbhjsGNwotkpOnHLnOdREJObBOKFIHIWrmkVvYtOPbqkwHkJISTxEeVGoEpYxCVZLxHYIeVZEvYKIRAphrEjjPLXhzjkNmpDTFmygPxjdnlsMewIEzkaInXJNVLydCKCxXPKNNrbMdNsFUCLOsTckPLfdqgRVySKiWAuLarrIiyInxkrEhwEaJOPqOViEciiddjKGZMXtrxwzhazwMOdQYQPtfaVxmfEUPSqerUavxybImhlqMbsGHvVSxsjHcjDtUsZOiFdYslxcPznexSuQclPzimsyVpRsBETNbEeMswKfBsAWIHROWDqqcWDCxMAACJIWsZiCGShNMvTMjckercodsKTetIsRlUkrshhySxkaYecWzzXvdaNisaMXikHgDQGUsDThPalhcZRxOUFgvYVQBBxAGOKDqiQZbELEfCjMlHpgWXxkPTpowGfuQBmkbkaMKLRPqoCifhrrSEnEXThgeSstayWjCDzObzopQHfZjHJyfnUTnGVphEMmKAoaLGvYWTuVuNhrDeHrEEzuFJRTXOcpoXoupQthUuxBZMDfleWNsbEQiOVJxMrInMKWTjENJjNPTjYDqwODnAeOwCEpjnPZMuXFhTsKUxODUyjGxwMzGAFSUsFnxNIXakyuwcEObgTjINISNxpDjZFRUIyFvTRQOgRisKfCyIzjHvavYGMXtAfLQBbqVrUqaSOPrnRzYEcikWUDOACfLeSfakDeXmvrvujXqNYTseNhgwDDROMXbYoWWEFqrOumFxbMrvpUAAwSGTZcvwhAGtxcasynOdXljvhNSLZcJseJNlkLXaZplPutdjYmzeKypqhnvvygHXclITfPFFgFQjsskwOKVPTVECERyZtiyRORBGXtsffoZznUdJKtLDpLStTQaRWWCIEfMblYUZWEwJTQEunzsaVaSPwPXImAGbQvKHAPyThtdsHtzISXIdjLGSTpfEiftZfMFFfYtuJDjOloUIvfCYYzokGdasZRRqywPrGgZZkqDqcUlcQhClvfLxxfhpOQLxNbIgYZTXLmqEbVyorAjjSRQjtediVObrcMPCvuPaLYjVfyetSkauaaPaIBBFhRDcpWSSgCmlZZslVLwLtpsFZsFGlekfjsiOuAeKskESEKDTHecktYWTiVAdOSulfYJkOhUTIrPcCdGlxRJTqtiCWjPTqGmwYiljfhWAlbQBrfpJDgRvIvWVHPtsbirJaOeHSAAgbxZMiuOjpFLsPIwMxRULRAMbtopZXhDAulWYcqBYASWoUagHyjODqyngsXaHDQkknNrWYiqspDqLqePQINLByFIDJfwUxhKIseYcWWgDoLmOJdYjnOXfVeGeUGQHubvbfickuQsfAWqxnMXZTVoTWEja"}'),
    (2, 1, '{"key": "really long value YAqvPiJOvePFCdZnExgTdHBBcergSanZurWvMuAKHkrlOQMbcahAZRbnhNSNDzPbzSUYOHPkkBekvGJSljwupfqvJYeDxFrrAQuWLWrmTbeJXwEDIXPsFsllOZiRhoZZfZkywitwXjjXbctpFmvdWSHGfhDKHXAVTaAmKHixUrsffAIEHyOhABNbactkOyaPAESfeBNjLrplctFzbFGyncGPCLEuRGLRvHPJTxoLOflcpAhNXmBpKRlUzFAoqwcJtEVUDsMSKegeoZDrwKAyalowYeKehFrBYVhQwUShEpQVUmhgMMhyBKXgPGUXcvbmbzzMXJCOdldKevErxATquWFRyAyRnPqJpgGTEYWGpGrbsytMPCLUQcqypdppbstKkkXNHiSTlivUHdUacTKZgYlZBlyVNuXQMlNpreNFSlSZQeyUPUJWPRrjXZcVoOWqccrXHevlYlTfvUKjSumqkeEdDJbnisSdfAFsbnmFqUUSopSMQYCmqTzlNbSGNIkBpPgQGpekMpjzQgVxLRcrIMxLszziUVlaHhjuHuPJrRFFySUxGibqJfNShvYRNOCDwduqigIabefDLulCBqQarndukzHpusvzRNaAdaOCtjTkdYNyxIPrvYkovSATrdrggwqqmtyygawARgaSCQJJbqpUNPgZynUdUDQUCqGkVeXQpFXHgAdoiOoqyksAarhWMLTFaBcuCNPvSKObFRGgGpGRaZxcteXEUspEkvcAFPnMnEQLAQBNjZdAtpbeDCMqnwCukjZhewVRxXBIGltuJOmguNFLmxpNbShcWyfHuQWTfikbJViXZowbPCmiDTJAtPWPuGwygeFqvbNHOydsBfvLLAZotkBgvprQVPJXqikUEngnKzeDmHLsLhWuHaSsvGArGHOMFOwwDbxUZnwAYRzwUhhVLWrXUrBJqSgYbYorWLVORokAJVkymvRUSYuBDfWgfSAMOncsNJpQhUnMofETXVAvcOudRmAumqVHxYdYgVhXbbYvMpkICHdwLoTnqNhmFuKHASmIDGkTdTvrBSTseOzBRirkSrYbRUHPImDqAWdxzHQBBpfhBBtYGEewoYTgKUGjawZHvPUOmFgbvkAJOWIGpEtqfMeiwgAsXEaKrAglnUiDGvSbvrPUCThSlvzNNhxiwHLXNIzpHdVOITySjgoTdTfVwuIEmATuJWKAOVuHCsauEcruUhenSeMQUohbMNpqrhIQqsLOyiMJFDPYeKkqYYfaYxwsoFpJjpzUSnWCRNyFAKXfCoTUILbtQXQdmxWrdNkelgImclanHcVwBTVqcZAloPwSpeXAZQbTxSmxixESUmtNgBxTRIMaxqUMoQWBYxCNmXumybtcPTjlevxeBlVxJNcoCfuHoFsKfaAbnCDjFQmHnmyvGQCSnGpxnMvvpyveTjlanbUaEoNQVzvGtirroPVpcFhIMcsWXdXjfyeqNxdPKqDeFUjTVcdPpIklQeEyypfHIkYzdZWqOdzYImiZCaAiDMHzjgDJeSLdugoGcDKrdRFsNGsYvTXtrDbAiVqqqyDodrJRehnxUjTRcdMHwONvUBMuszdHuCCOAiaiYGKcfDQWnSCHwItTGSfJuZBSINipyUNEjMxNQqLbBAozILkZyGmyaltHIFtiSfsfXuOsfhnsJhqXEKNZveVDTTJNJVVovhPmjDXrsQayJVKUOTxdjPkjblfehLleQveAWMAPMOQGrTMrhKNtJdRFYcpIOWsrXJdJQZEfqxBVtuzbfOUEvwChOxHJGViQIayngcoaaWQhNDjUvNdaNjyPEHbhfMThOsxcrEGgATMOmdZYahdmnbVNjBGVyzFZQGWOmiAsJAuwSzLmdNBXlriIRYcHBGSZahIjDMJdfEYQFKfVoyLflVdJwzMVXmeVqVZgaZgcehQTQyKJQiuFBKWRseBRGGTOFfTsfQotIvyOrUpAEeNtiucRmpMyUkoYaTlwJumciEiJlHMHRsmwcwgDbPxVLnGuhxjxFmhogidgoDFqkKhTPQxVYdpTPChuugbcDUwqjBOItoADBQppUkwhGEHebqFAbuPcdcSTyvWLCWUhgCmhUnCEbuURbLOrVcUsDYdLQdoflcssuNhRCROXFqAxRlaBsAtrPLkSliDvDkVNJqQaqUYrTAGocwVZpJYslwiwwovrZyKtCLyGHQDhGjZPVDWyLufRmKebrTnkEPmmbGxVjrCQZIHQfTKjHzZbxErRdlXBkEfJlPDEQSzVGWacLFcnsDdEIzGFzmiEmAjmQquCIUvYTdcamrcJzwYDkTxMiyiAfGOpPnxWOrUzayhrzEJaXIXWWNKLTNEBcfdQjsMdenIHjqLhEhmQBWBsSDAuoIdOsjUikxqdRVSyjJcjHoXEsssrLTbTlCuCvMXaDJUdWkjKfvLRuFmdPFbolxrJSzWdYTtTHoLSWtVLjDIMZDXRtwDvlJYXRPCyUijGKBpBStaeGwPUZOVjdOFGdFJrOJwAXLlKdRTVGghpGJvxNlNvxQkejnHJKOyAMuiSkfPCHVQiumnKgguRXxruSuRHHGSWoStPLrORvsHcWQGXPOoLwfpgnUJaNnzkJEWMKwkfdYQCckzAjyxzLChdupSvgkldxxhPRJZEqsjpIztYNEBJWTJzeNkaCARWxtYhrwwGUysarjCjojQsubLTgSsLKzGYyYSwGooUYrhNvkuLCturXnVHFJnaHIUtFehaSRznlrDYXfuECSUvITJXzsOBqpDzkWSBnzrwQpaxWbMsSHPUzNZqRHGdzNXsPzNKkEsYjFeFdTolAAwmzVNpKWigDQhIQXtpDhHCPonSXbIsquykxguffcmsOWbUPGMTHteNUmLOGEnCgWctZaIDmXKrgPcQgqyeuUFpuJSflOwsPSCBVqSWyeNALjfbYPHWxYJPzBkTYPHVdhiBZGqZpPnNMSdSnXMNDQxsVuxescdIdbKiHxfCZeqjmUenSQpZbloFvYYbKiTEdwJrvXrHVmxv"}');

      Now, start the connector.

      After the initial snapshot completes, do some updates on the db rows:

      UPDATE test_table
SET x = 2;
UPDATE test_table
SET x = 3;

      The connector crashes:

      2018-07-27 19:54:09,619 ERROR  Postgres|my_test_database|records-stream-producer  unexpected exception while streaming logical changes   [io.debezium.connector.postgresql.RecordsStreamProducer]
org.postgresql.util.PSQLException: ERROR: no known snapshots
  Where: slot "debezium", output plugin "wal2json", in the change callback, associated LSN 0/16D3850
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
        at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1043)
        at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:962)
        at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:41)
        at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:145)
        at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:114)
        at org.postgresql.core.v3.replication.V3PGReplicationStream.read(V3PGReplicationStream.java:60)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:198)
        at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
        at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

      Errors are also visible in the db server logs:

      2018-07-27T19:54:09.616547179Z ERROR:  no known snapshots
2018-07-27T19:54:09.616587252Z CONTEXT:  slot "debezium", output plugin "wal2json", in the change callback, associated LSN 0/16D3850

      This is a rather contrived example, but we have also encountered this problem with other seemingly harmless rows in our database. We suspect that this is related to wal2json's handling of TOAST data. (The long strings I used in my example also happen to be highly incompressible.)

      wal2json versions prior to ce82d7387a806bc8bacaeed80e0a0f1840733dce omitted unchanged TOAST columns from the change event. Newer versions include all changes, but this may have introduced a regression in Debezium, or in the wal2json plugin itself.

              jpechane Jiri Pechanec
              csang1 Christopher Sang (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: