Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5133

Ratelimiter logger for MySqlValueConverters

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Unresolved
    • Major
    • 2.2-backlog
    • None
    • mysql-connector
    • None
    • False
    • None
    • False
    • 0
    • 0% 0%

    Description

      Some logging, can be particularly strong perf hit, for example all of the ones that have to be logged for every row.

       

      E.g. In my case I have a column that for historical reason has a lot of 0000-00-00 00:00:00, which is an invalid value for most DB, debezium included:

      https://github.com/debezium/debezium/blob/main/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/MySqlValueConverters.java#L864

       

      
              if (year == 0 || month == 0 || day == 0) {
                  LOGGER.warn("Invalid value '{}' stored in column '{}' of table '{}' converted to empty value", timestampString, column.name(), table.id());
                  return true;
              }
      
      

      In my case, this gets logged up to 50-100 times per MILLISECOND, and tens of thousands per second, consuming basically all cycles

       

       

      A solution such as:

      https://github.com/Swrve/rate-limited-logger#sample-output 

       

      that ratelimited even at levels such 50-300/s, would already be a great improvement.

      
              if (year == 0 || month == 0 || day == 0) {
                  CONVERSION_LOGGER_RL.warn("Invalid value '{}' stored in column '{}' of table '{}' converted to empty value", timestampString, column.name(), table.id());
                  return true;
              }
      
      

       

      Would you agree in adding this?

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            mark4912 Mark Mark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: