Uploaded image for project: 'Observability and Data Analysis Program'
  1. Observability and Data Analysis Program
  2. OBSDA-58

Prometheus Rule for ElasticsearchCluster and Kibana Healthchecks

XMLWordPrintable

    • False
    • False
    • Undefined

      Hi Team,

       

      oc v3.11.216
      kubernetes v1.11.0+d4cacc0

       

      The customer has raised concern to introduce the rule to monitor the elasticsearch indices as they were getting 504 Gateway Time-out error on Kibana with the large size of indices. 

       

      History of the issue:-

      The customer reported a case with us by saying that they are getting 504 Gateway Time-out error on Kibana 

       

      We were able to see that elasticsearch health is fine and all the pods are running up and fine.

       

      The below log shows elasticsearch is in the green state.

      ~~~

      epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1614035597 23:13:17 logging-es green 1 1 751 751 0 0 0 0 - 100.0%

      ~~~

       

       - While checking more into the issue we increased the elasticsearch memory but it still won't make much difference.

      • At last, the customer deleted the large indices after which everything is working fine.

       

      Customers need a solution so that they will get an alert before the time for the indices with large sizes.

       

       

              rh-ee-rfloren Roger Florén
              rhn-support-aharchin Akhil Harchinder (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: