Prometheus Rule for ElasticsearchCluster and Kibana Healthchecks

Hi Team,

oc v3.11.216
kubernetes v1.11.0+d4cacc0

The customer has raised concern to introduce the rule to monitor the elasticsearch indices as they were getting 504 Gateway Time-out error on Kibana with the large size of indices. 

History of the issue:-

The customer reported a case with us by saying that they are getting 504 Gateway Time-out error on Kibana 

We were able to see that elasticsearch health is fine and all the pods are running up and fine.

The below log shows elasticsearch is in the green state.

~~~

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1614035597 23:13:17 logging-es green 1 1 751 751 0 0 0 0 - 100.0%

~~~

 - While checking more into the issue we increased the elasticsearch memory but it still won't make much difference.

• At last, the customer deleted the large indices after which everything is working fine.

Customers need a solution so that they will get an alert before the time for the indices with large sizes.