Story

As a log console user I want to select logs to be viewed using the kubernetes label and field selector syntax, described here:
[Labels and Selectors | Kubernetes|https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/]

Rationale

• This syntax is used by the kubectl and oc tools, and is familiar to kubernetes and openshift users.
• The logging console is primarily for viewing k8s logs, and should be useful regardless of the choice of back-end store.
• Users should be able to use the console without learning  a new query langauge.

Acceptance Criteria

• User can enter the same label selectors that they use with kubectl to select pods for viewing logs in the console.
• User can enter  field selectors for universally selectable fields (e.g. metadata.name) to select pods for viewing (see below)
• User can also use logQL queries to select logs
• On first opening the log console, k8s selector feature is prominent but  loki queries are easy to find.
• k8s selectors are always available regardless of log store back-end. Back-end specific queries (Loki, in future Elasticsearch...) are available only when relevant.

Notes

Minimum requirements are above, this section includes "nice-to-have" features and implementation notes.

Query Syntax

k8s selectors are a bit haphazard in their syntax and capabilities.

• The oc syntax only allows logical AND expressions (with a comma) 
• Selectors can only choose pods, but the unit of logging is the container. Selectors can't pick specific containers.
• Field selectors are restricted to different fields for each resource type.
  • Some fields are always selectable, notably metadata.name and namespace (these are also available in the log record)
  • Some selectable fields may not be included in the log record, so the console may be more restricted than kubectl here.

We could provide some simple but useful extensions to the syntax:

• Include a phony "label" name that selects by container names
• Allow multiple selector strings (e.g. separate text boxes), where the terms in each comma-separted selector string mean "AND", but separate selector strings are "ORed". A bit like  multiple queries in the Prometheus metric console.

Query translation

The log collector copies k8s metadata into Loki labels and JSON fields in the log record.

Some translation is needed between names in k8s selectors and Loki form: e.g.:

• ks label app.kubernetes.io/name => 
  • JSON field kuberntes_labels_app_kubernetes_io_name
• k8s field metadata.name => 
  • JSON field kuberntets_pod_name
  • loki label pod_name

To translate label and field selectors into a loki query we need to:

• Map k8s label and field names to their respective loki label or JSON field names.
• Prefer loki labels over JSON filters if both are available
• Do some boolean normalization & separation of terms into Loki labels and JSON fields to generate the Loki {selector} and set of Loki |= filters.  

Open questions

• Can we combine field and label queries into one string, or should we separate them? If combined we need to deal with possible label/field name clashes.
• If we add the ability to select by "container", should that be a phony field selector or a phony label selector?
• Can we remember the users preference for loki or k8s queries and show just the preferred query box by default?