Task

Review & update the internal data model documentation and organization.

Goals

• Treat the data model as a contract or protocol between us and our users, and between our own components: collector, forwarder, normalizer, and store. 
• Remove unused fields
• User Documentation: Generate documentation of the expected format of log records.
  • Establish a process for the doc team to directly update the source
• Internal Documentation: Superset of user documentation, includes internal/private fields
• Classify fields as static or dynamic
  • Static values are the same for every record in a log stream (for example Container-ID never changes over a container's life)
    They can sometimes be encoded as part of a stream-id or attributes to avoid sending on every record: Loki labels, CloudWatch stream IDs, etc.
  • Dynamic values may be different in every record, so they must be sent each time.
• Multi-encoding: The model can be encoded in multiple ways, add notes on mapping to JSON, Loki labels, Syslog STRUCTURED-DATA, etc.

Acceptance Criteria

• User documentation can be generated automatically from the model.
• A documented process for the doc team to update the model source.
• Removed out-of-date entries
• Classified remaining entries: static/dynamic, public/private