Requirements and Acceptance Criteria

• Add input formats to HTTP receiver, including at least:
  • ndjson - newline-delimited JSON records, one record per line.
  • array - HTTP body is JSON array of log records
  • array.path.to.items - HTTP body is JSON object with array field 'path.to.items'
  • text - newline delimited plain text as would be scraped from a file
  • kubeAPIAudit - a k8s audit event list, equivalent to: array.itemsk
• Support other log types: node audit logs, application, infrastructure
• Update API if required

Use cases

As a logging admin I want to

1. Receive Viaq-format logs from openshift-logging on another cluster and forward them as they would have been forwarded if collected from the current cluster.
2. Receive kube API audit logs and forward them as audit logs.
3. Receive unknown JSON or text-format log records and forward them as "unknown" logs
4. Add fields to received logs to identify their origin. Such fields may or may not be part of the viaq model. User is responsible for creating valid log records.

Design summary

We will add a 3rd basic log type unknown to [infrastructure, audit, application]

The receiver extracts log records from HTTP request bodies according to its format

Next it adds fields to the internal log record from the "origin" field 

TODO need more API.

Open Questions

Record types: object only or accept plain string?

We need TLS for the  service, see https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/

Can we do the TLS  usign k8s outside the collector, or do we need internal TLS & certificate setup?