Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3860

The chunk select statement is incorrect for combined primary key in incremental snapshot

XMLWordPrintable

      Code position : SignalBasedIncrementalSnapshotChangeEventSource#buildChunkQuery

      @Test
      public void testBuildQuery() {
          final SignalBasedIncrementalSnapshotChangeEventSource<TableId> source = new SignalBasedIncrementalSnapshotChangeEventSource<>(
                  config(), new JdbcConnection(config().getConfig(), config -> null), null, null, SnapshotProgressListener.NO_OP, DataChangeEventListener.NO_OP);
          final IncrementalSnapshotContext<TableId> context = new IncrementalSnapshotContext<>();
          source.setContext(context);
          final Column pk1 = Column.editor().name("pk1").create();
          final Column pk2 = Column.editor().name("pk2").create();
          final Column val1 = Column.editor().name("val1").create();
          final Column val2 = Column.editor().name("val2").create();
          final Table table = Table.editor().tableId(new TableId(null, "s1", "table1")).addColumn(pk1).addColumn(pk2)
                  .addColumn(val1).addColumn(val2).setPrimaryKeyNames("pk1", "pk2").create();
          Assertions.assertThat(source.buildChunkQuery(table)).isEqualTo("SELECT * FROM \"s1\".\"table1\" ORDER BY pk1, pk2 LIMIT 1024");
          context.nextChunkPosition(new Object[]{ 1, 5 });
          context.maximumKey(new Object[]{ 10, 50 });
          Assertions.assertThat(source.buildChunkQuery(table)).isEqualTo(
                  "SELECT * FROM \"s1\".\"table1\" WHERE pk1 >= ? AND pk2 >= ? AND NOT (pk1 = ? AND pk2 = ?) AND pk1 <= ? AND pk2 <= ? ORDER BY pk1, pk2 LIMIT 1024");
      }
      

      Just imaging following case, assuming chunk size is 3, and the second chunk will never 

      be selected due to it's pk2  column is smaller than the pk2(110) end of chunk-1 

      pk1 pk2
      1   101
      2   105
      3   110
      4   1    
      5   2
      6   3

       

       

       

              jpechane Jiri Pechanec
              leonardxu leonard xu (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: