Details
-
Feature Request
-
Resolution: Done
-
Major
-
1.1.0.Final
-
None
Description
Context
- Debezium uses Log4j for logging and currently logs stacktraces over multiple lines.
- As a result some infrastructure components for monitoring can not link the different lines of the stacktrace together and interpret every line as a new log event.
- Therefore, monitoring alerts filters often miss required context to determine whether an alert should be send.
Example
Debezium 1.1.0.Final introduced a retry-mechanism for lost Postgres connections but the following is still logged when a disconnect occurs
2020-03-27 07:52:43,168 ERROR Postgres|audit|postgres-connector-task Producer failure [io.debezium.pipeline.ErrorHandler] org.postgresql.util.PSQLException: Database connection failed when writing to copy at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1052) at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23) at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:193) at org.postgresql.core.v3.replication.V3PGReplicationStream.timeUpdateStatus(V3PGReplicationStream.java:184) at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:126) at org.postgresql.core.v3.replication.V3PGReplicationStream.readPending(V3PGReplicationStream.java:80) at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:402) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:125) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:101) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) at org.postgresql.core.PGStream.flush(PGStream.java:554) at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1050) ... 13 more
Since Debezium will reconnect to Postgres, no alert should be send since Debezium automatically recovers from the issue. However, the SocketException could still be interpreted as an issue to alert on since it misses the context of it occurring for a Postgres reconnect.
Requested change
- The possibility to log the different lines of a stacktrace as a single log event. E.g. by allowing to log in JSON format.
- This is supported out-of-the-box by Log4j2 but not by Log4j (1): https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout
- Alternatively:
- There are solutions using Log4j (1) + Logback: https://docs.datadoghq.com/logs/log_collection/java/?tab=log4j
- There are log4j extensions that could provide the functionality: https://github.com/logstash/log4j-jsonevent-layout
Note
- We are using Debezium through the provided Docker container so we'd like to configure the functionality as mentioned in https://debezium.io/documentation/reference/operations/logging.html#configuring-log-level-docker
