Use Grafana and other tools to examine OVN-K metrics

Use Grafana and promql to observe/measure OVN-K metrics extracted from an OCP prometheus db. Note: this has to be limited in scope because I don't have the tools or skills to drive this down to the level of determining key metrics for customer performance issues or scale headroom. I think that is a Perf & Scale team ask.

The big idea:

• A grafana dashboard for extracted Prometheus metrics
• A grafana dashboard for extracted metrics plus the results of must-gathers and sosreports
• A grafana dashboard for a running cluster in OCP

To start the grafana container, pointing to an existing prometheus database:

[root@openshift-jumpserver-0 tmp]# datadir=/tmp/prometheus-2.24.1.linux-amd64/data
[root@openshift-jumpserver-0 tmp]# echo $datadir
/tmp/prometheus-2.24.1.linux-amd64/data
[root@openshift-jumpserver-0 tmp]# podman run --rm --privileged -p 3000:3000 -p 9090:9090 -v $datadir/:/data:Z  quay.io/distributed-systems-analysis/prom-graf-visualizer:latest

• To set up the local SSH tunnel that allows me to point my browser to
  http://localhost:3000 and connect to the Grafana container:

  cfields-mac:~ cfields$ ssh -p 8022 -L 3000:127.0.0.1:3000 -L 9090:127.0.0.1:9090 root@10.1.164.9 -N
  

• examples of 4 queries that should have data, as collected by prometheus-metrics-dump

  podman exec -it romantic_jang /bin/bash
  ./promtool query instant http://localhost:9090 '{__name__=~"ovn__.*"}[2d]'
  ./promtool query instant http://localhost:9090 '{__name__=~"ovnkube_.*"}[2d]'
  ./promtool query instant http://localhost:9090 '{__name__=~"pod_.*"}[2d]'
  ./promtool query instant http://localhost:9090 '{__name__=~"etcd_.*"}[2d]'