Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6155

Oracle path used current batchSize to calculate end scn is wrong, need to use min batch size

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Minor
    • 2.1-next, 2.2.0.CR1
    • 1.9.1.Final, 2.0.0.Final
    • oracle-connector
    • None

    Description

      private Scn calculateEndScn(OracleConnection connection, Scn startScn, Scn prevEndScn) throws SQLException {

          .....

          Scn currentScn = archiveLogOnlyMode? getMaxArchiveLogScn(currentLogFiles)
              : connection.getCurrentScn(); streamingMetrics.setCurrentScn(currentScn);

          // Add the current batch size to the starting system change number
          final Scn currentBatchSizeScn = Scn.valueOf(streamingMetrics.getBatchSize());
          Scn topScnToMine = startScn.add(currentBatchSizeScn);

          // Control adjusting batch size
          boolean topMiningScnInFarFuture = false;

       

         // this condition mathematical expresion as below show

          topScnToMine - currentScn >currentBatchSizeScn

          because topScnToMine = startScn + currentBatchSizeScn

          so  this condition will become startScn - currentScn > 0

          it's always false. we need compare it with defaultBatchSize

          if (topScnToMine.subtract(currentScn).compareTo(currentBatchSizeScn) > 0)

         

      {                 streamingMetrics.changeBatchSize(false, connectorConfig.isLobEnabled());         topMiningScnInFarFuture = true;     }

          }

          // same reason, we need compare it wih defaultBatchSize

          if (currentScn.subtract(topScnToMine).compareTo(currentBatchSizeScn) > 0)

      {         streamingMetrics.changeBatchSize(true, connectorConfig.isLobEnabled());     }

      ......

      }

       

       

      The reason is:

       

      1. topScnToMine.subtract(currentScn).compareTo(currentBatchSizeScn) > 0 result always is false, because topScnToMine = startScn.add(currentBatchSizeScn), so while currentBatchSizeScn increment at max will nevent reduce. it is a issue, need to fix.
      1. currentScn.subtract(topScnToMine).compareTo(currentBatchSizeScn) > 0 change to compare with default reason is that i think if currentScn - topScnToMine = currentBatchSizeScn meaning lag is increasing, even if we can catch up to currenScn next time. if compare with defaultBatchSize, in the worst case there is only a lag of defaultBatchSize size.

      Attachments

        Issue Links

          Activity

            People

              vjuranek@redhat.com Vojtech Juranek
              doupengwei pengwei dou (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: