Uploaded image for project: 'OpenShift Request For Enhancement'
  1. OpenShift Request For Enhancement
  2. RFE-3898

Improve scrape sample alerts

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • None
    • Documentation, Monitoring
    • None
    • False
    • None
    • False
    • Not Selected

      Our documentation suggests creating an alert after configuring scrape sample limits.

      That PrometheusRule object has two alerts configured within it [1]

       

      `ApproachingEnforcedSamplesLimit` 

      `TargetDown` 

       

      The `Targetdown` alert is designed to fire after the `ApproachingEnforcedSamplesLimit` because the target is dropped once the enforced sample limit is reached

       

      The TargetDown alert is creating false positives - its firing for reasons other than pods in the namespace have reached there enforced sample limit. 

      This has been discussed with the monitoring team and the suggestion was to raise this RFE to investigate the possible use of the following Prometheus feature flag  [2] in order to improve the quality of this alert 

      [1] - https://docs.openshift.com/container-platform/4.12/monitoring/configuring-the-monitoring-stack.html#creating-scrape-sample-alerts_configuring-the-monitoring-stack 

      [2] - https://prometheus.io/docs/prometheus/latest/feature_flags/#extra-scrape-metrics

            sstout@redhat.com Stephanie Stout
            rhn-support-nigsmith Nigel Smith
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: