Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-11031

Json encoding of Access Log Events

XMLWordPrintable

      Request

      It should be possible to emit access log events json encoded to allow log aggregation systems to ingest pre-parsed events for filtering and analytics.

      Background

      Application logs can be emitted json encoded allowing log aggregation systems like logstash/ELK to ingest and parse the events. This is especially important when working with containers, but anything requiring log aggregation would benefit. Not all log aggregation systems allow after-the-fact parsing of log events (e.g. Kibana) and developers don't have access to the log aggregator to install a parser like logstash/filebeat.

      Access log events, however, can't be json encoded. The whole log event can be, but what's important about access log events can't.

      Example result

      There are many ways to encode the event into json. Ideally it would be configurable. One way would be to use message, but instead of it just containing a String it could contain json. However, abusing the message field might not work because it is of type String|text. So even if it contained valid json, it wouldn't be processed as such. Separate fields might need to be used. Using message is probably preferred because if not, what else would that field contain?

      ...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": {
  "x-forwarded-for": "-",
  "remoteHost": "172.20.2.1",
  ...
}
...
# alternatively
"x-forwarded-for": "-",
"remoteHost": "172.20.2.1",
"remoteUser": "200",
"dateTime": "[14/Sep/2018:20:59:37 +0000]",
"requestLine": "GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0",
"status": "200",
"bytesSent": "81862",
"timeTaken": "10",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
...

      Attempted Workaround

      Making the pattern look like json failed due to double quote escaping:

      <access-log pattern='{\"x-forwarded-for\":\"%{i,X-Forwarded-For}\",\"remoteHost\":\"%h\",\"remoteUser\":\"%s\",\"dateTime\":\"%t\",\"requestLine\":\"%r\",\"status\":\"%s\",\"bytesSent\":\"%b\",\"timeTaken\":\"%D\",\"userAgent\":\"%{i,User-Agent}\"}' use-server-log="true" />

      Result

      ...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": "{\\\"x-forwarded-for\\\":\\\"-\\\",\\\"remoteHost\\\":\\\"172.20.2.1\\\",\\\"remoteUser\\\":\\\"200\\\",\\\"dateTime\\\":\\\"[14/Sep/2018:20:59:37 +0000]\\\",\\\"requestLine\\\":\\\"GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0\\\",\\\"status\\\":\\\"200\\\",\\\"bytesSent\\\":\\\"81862\\\",\\\"timeTaken\\\":\\\"10\\\",\\\"userAgent\\\":\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\\\"}",

      Json formatting log configuration

      <subsystem xmlns="urn:jboss:domain:logging:5.0">
  <console-handler name="CONSOLE">
    <level name="DEBUG"/>
    <formatter>
      <named-formatter name="JSON_FORMATTER"/>
    </formatter>
  </console-handler>
...
  <formatter name="JSON_FORMATTER">
    <json-formatter>
      <exception-output-type value="formatted"/>
      <key-overrides timestamp="@timestamp" loggerName="logger_name"/>
      <meta-data>
        <property name="@version" value="1"/>
      </meta-data>
    </json-formatter>
  </formatter>
...

              jperkins-rhn James Perkins
              fschmager Frank Schmager (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: