Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-11031

Json encoding of Access Log Events

    XMLWordPrintable

Details

    Description

      Request

      It should be possible to emit access log events json encoded to allow log aggregation systems to ingest pre-parsed events for filtering and analytics.

      Background

      Application logs can be emitted json encoded allowing log aggregation systems like logstash/ELK to ingest and parse the events. This is especially important when working with containers, but anything requiring log aggregation would benefit. Not all log aggregation systems allow after-the-fact parsing of log events (e.g. Kibana) and developers don't have access to the log aggregator to install a parser like logstash/filebeat.

      Access log events, however, can't be json encoded. The whole log event can be, but what's important about access log events can't.

      Example result

      There are many ways to encode the event into json. Ideally it would be configurable. One way would be to use message, but instead of it just containing a String it could contain json. However, abusing the message field might not work because it is of type String|text. So even if it contained valid json, it wouldn't be processed as such. Separate fields might need to be used. Using message is probably preferred because if not, what else would that field contain?

      ...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": {
  "x-forwarded-for": "-",
  "remoteHost": "172.20.2.1",
  ...
}
...
# alternatively
"x-forwarded-for": "-",
"remoteHost": "172.20.2.1",
"remoteUser": "200",
"dateTime": "[14/Sep/2018:20:59:37 +0000]",
"requestLine": "GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0",
"status": "200",
"bytesSent": "81862",
"timeTaken": "10",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
...

      Attempted Workaround

      Making the pattern look like json failed due to double quote escaping:

      <access-log pattern='{\"x-forwarded-for\":\"%{i,X-Forwarded-For}\",\"remoteHost\":\"%h\",\"remoteUser\":\"%s\",\"dateTime\":\"%t\",\"requestLine\":\"%r\",\"status\":\"%s\",\"bytesSent\":\"%b\",\"timeTaken\":\"%D\",\"userAgent\":\"%{i,User-Agent}\"}' use-server-log="true" />

      Result

      ...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": "{\\\"x-forwarded-for\\\":\\\"-\\\",\\\"remoteHost\\\":\\\"172.20.2.1\\\",\\\"remoteUser\\\":\\\"200\\\",\\\"dateTime\\\":\\\"[14/Sep/2018:20:59:37 +0000]\\\",\\\"requestLine\\\":\\\"GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0\\\",\\\"status\\\":\\\"200\\\",\\\"bytesSent\\\":\\\"81862\\\",\\\"timeTaken\\\":\\\"10\\\",\\\"userAgent\\\":\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\\\"}",

      Json formatting log configuration

      <subsystem xmlns="urn:jboss:domain:logging:5.0">
  <console-handler name="CONSOLE">
    <level name="DEBUG"/>
    <formatter>
      <named-formatter name="JSON_FORMATTER"/>
    </formatter>
  </console-handler>
...
  <formatter name="JSON_FORMATTER">
    <json-formatter>
      <exception-output-type value="formatted"/>
      <key-overrides timestamp="@timestamp" loggerName="logger_name"/>
      <meta-data>
        <property name="@version" value="1"/>
      </meta-data>
    </json-formatter>
  </formatter>
...

      Attachments

        Issue Links

          is blocked by

          Feature Request - Feature requests from customers and/or users WFCORE-4355 Add utility for event logging such as audit and access logging

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Task - Represents a small unit of work that is not end-user facing. WFLY-11816 Bump the Undertow subsystem model version and schema to version 9

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is related to

          Feature Request - Feature requests from customers and/or users HAL-1600 Expose ability to configure the JSON encoded access logs

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Open

          Activity

            People

              jperkins-rhn James Perkins
              fschmager Frank Schmager (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: