Create an ansible installer for Prometheus Server

Notes from emails :

The guide is under works. However Joel Diaz from OpenShift Online team has been trying it out and here is what he sent me. I know he is super-busy, but, maybe able to help you with some additional questions or if you get stuck trying this.

I don't have any specific documents on installing Prometheus, but the process is basically:

0) create a custom alert file that looks something this:
groups:

• name: example-rules
  rules:
• alert: "node_down_alert"
  expr: up {job="kubernetes-nodes"}

  == 0
  annotations:
  miqTarget: "ContainerNode"
  severity: "HIGH"
  url: "https://www.example.com/fixing_instructions"
  message: "kubernetes-nodes job is down on $labels.instance"

• alert: BigMemPod
  expr: container_spec_memory_limit_bytes {job="kubernetes-cadvisor", container_name!=""}

  > 15000000000
  annotations:
  miqTarget: "ContainerNode"
  severity: "HIGH"
  message: "Huge pod $labels.container_name detected"
  url: "https://www.example.com/fixing_instructions"

1) add the following variables to your openshift-ansible cluster inventory file (the one you used to install the cluster):

openshift_prometheus_state=present
openshift_prometheus_node_selector=

openshift_prometheus_additional_rules_file=<PATH TO CUSTOM PROMETHEUS ALERT RULES>

2) install prometheus by running the prometheus installation playbook:

ansible-playbook -i <INVENTORY FILE> ./git/openshift-ansible/playbooks/byo/openshift-cluster/openshift-prometheus.yml

3) wait for the pod to be ready, and then you should be able to access prometheus through the created route on the cluster/namespace:

oc get routes -n prometheus

4) grant whatever user you are using to authenticate through the OAuth proxy 'view' access to the 'prometheus' namespace:

oc adm policy add-role-to-user view jdiaz@redhat.com -n prometheus

========================================================
If you just want a stand-alone version of prometheus, then you probably don't want all the extra pieces that the official openshift-ansible installer brings in. You probably want to create a custom deployment using the official prometheus docker images: https://prometheus.io/docs/introduction/install/ . I'm not sure what this instance of Prometheus would have permissions to scrape data from, but it should result in a stand-alone Prometheus server that just happens to be running as a pod on OpenShift.
========================================================

For this use case. I would suggest writing a new service under Service Catalog with Kubernetes OpenServiceBroker API and with back end Prom-service being implemented as a OpenShift Template or a OpenShift Ansible Broker (previously Anisble Service Broker).

So upstream OpenShift Origin document already has this. So u can start here [1].

Specifically, on OpenShift Ansible Broker (previously called Ansible Service broker and how referenced upstream, you can look at [2]).

In addition Eric Dube and Diogenes (PMs) and Paul Morie/John Matthwes (eng leads from RH) can help as well. Let me know if you have any questions.

[1] https://docs.openshift.org/latest/architecture/service_catalog/ansible_service_broker.html#overview
[2] https://github.com/ansibleplaybookbundle