Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8610

Add SnapshotSkipped status to metrics

XMLWordPrintable

      Bug report

      debezium_metrics_SnapshotCompleted is 1.0 in Prometheus JMX Exporter if debezium gets an offset from redis key

       

      "message":"Offsets: { [\"redis\",{\"server\":\"rdi\"}]={\"last_snapshot_record\":true,\"lsn\":311783848 ... 

       

      What Debezium connector do you use and what version?

       

      io.debezium.connector.postgresql.PostgresConnector 

      Debezium version 2.7.3.Final

       

      What is the connector configuration?

       

      # add additional location for mounted external secrets
      quarkus.config.locations=file:/debezium/mounted-properties/
      debezium.sink.type=redis
      debezium.sink.redis.message.format=extended
      debezium.sink.redis.address=${RDI_REDIS_HOST}:${RDI_REDIS_PORT}
      debezium.sink.redis.user=${RDI_REDIS_USERNAME:}
      debezium.sink.redis.password=${RDI_REDIS_PASSWORD:}
      debezium.sink.redis.ssl.enabled=${RDI_REDIS_SSL:false}
      debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
      debezium.source.plugin.name=pgoutputdebezium.source.offset.storage=io.debezium.storage.redis.offset.RedisOffsetBackingStore
      debezium.source.topic.prefix=rdi
      debezium.source.database.dbname=chinookdebezium.source.database.hostname=host.docker.internal
      debezium.source.database.port=5432
      debezium.source.database.user=postgres
      debezium.source.database.password=postgres
      debezium.source.include.schema.changes=false
      # This property takes effect when the `REPLICA IDENTITY` of the table is set to `FULL`.
      debezium.source.skip.messages.without.change=true
      debezium.source.offset.flush.interval.ms=1000
      debezium.source.tombstones.on.delete=false
      debezium.source.schema.history.internal=io.debezium.storage.redis.history.RedisSchemaHistory
      debezium.source.key.converter.schemas.enable=false
      debezium.source.value.converter.schemas.enable=false
      debezium.source.decimal.handling.mode=string
      debezium.source.table.include.list=public.invoice,public.track,public.mediatype,public.customer,public.genre,public.invoiceline,public.playlist,public.employee,public.album,public.artist,public.playlisttrack
      debezium.source.column.include.list=public.employee.state,public.customer.phone,public.playlist.playlistid,public.genre.name,public.invoice.invoiceid,public.employee.postalcode,public.invoiceline.quantity,public.employee.city,public.invoiceline.invoiceid,public.invoice.billingcity,public.track.milliseconds,public.customer.supportrepid,public.invoice.invoicedate,public.track.bytes,public.customer.country,public.track.unitprice,public.invoice.billingstate,public.invoiceline.unitprice,public.employee.firstname,public.invoice.billingcountry,public.employee.reportsto,public.track.trackid,public.employee.address,public.employee.email,public.customer.state,public.track.mediatypeid,public.mediatype.name,public.customer.customerid,public.playlist.name,public.album.title,public.customer.lastname,public.track.composer,public.employee.fax,public.playlisttrack.playlistid,public.genre.genreid,public.customer.address,public.employee.title,public.invoice.total,public.employee.employeeid,public.customer.email,public.album.albumid,public.employee.birthdate,public.track.name,public.invoiceline.invoicelineid,public.employee.country,public.invoice.customerid,public.invoice.billingpostalcode,public.customer.fax,public.track.albumid,public.customer.firstname,public.invoiceline.trackid,public.customer.postalcode,public.employee.lastname,public.employee.phone,public.mediatype.mediatypeid,public.customer.company,public.artist.artistid,public.track.genreid,public.playlisttrack.trackid,public.invoice.billingaddress,public.artist.name,public.album.artistid,public.customer.city,public.employee.hiredate
      debezium.source.message.key.columns=public.invoice:invoiceid;public.track:trackid;public.mediatype:mediatypeid;public.customer:customerid;public.genre:genreid;public.invoiceline:invoicelineid;public.playlist:playlistid;public.employee:employeeid;public.album:albumid;public.artist:artistid;public.playlisttrack:playlistid,trackiddebezium.transforms=AddPrefix
      debezium.transforms.AddPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
      debezium.transforms.AddPrefix.regex=.*
      debezium.transforms.AddPrefix.replacement=data:$0quarkus.log.min-level=TRACE
      quarkus.log.level=DEBUG
      quarkus.log.console.json=true
      quarkus.log.console.json.key-overrides=logger_name=logger_name,stack_trace=exc_info
      quarkus.log.console.json.exception-output-type=formatted
      quarkus.log.console.json.excluded-keys=loggerClassName,sequence,hostName,mdc,ndc,threadId,threadName,threadPriority
      quarkus.http.port=8085# Log handler for termination-log. This is picked up by K8S to indicate container status lastState.message field
      quarkus.log.file.level=ERROR
      quarkus.log.file.json=false
      quarkus.log.file.enable=true
      quarkus.log.file.format=%s
      quarkus.log.file.path=/dev/termination-log 

       

      What is the captured database version and mode of deployment?

      https://github.com/mgutz/postgres-chinook-image - postgres database sample database chinook docker image

      What behavior do you expect?

      This is the behaviour from the start when there is no offset:

      # HELP debezium_metrics_SnapshotCompleted debezium.postgres:name=null,type=connector-metrics,attribute=SnapshotCompleted
      # TYPE debezium_metrics_SnapshotCompleted untyped
      debezium_metrics_SnapshotCompleted{context="snapshot",name="rdi",plugin="postgres"} 1.0

      What behavior do you see?

      When there is offset:

      # HELP debezium_metrics_SnapshotCompleted debezium.postgres:name=null,type=connector-metrics,attribute=SnapshotCompleted # TYPE debezium_metrics_SnapshotCompleted untyped debezium_metrics_SnapshotCompleted{context="snapshot",name="rdi",plugin="postgres"} 0.0

       

      Connector logs

      debezium.logs.txt.gz

       

       

              Unassigned Unassigned
              wheelly Boris Kolesnikov (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: