Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-1708

Enforce label scrape limits for UWM

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Normal
    • openshift-4.11
    • None
    • None
    • False
    • False
    • NEW
    • Undefined
    • Sprint 216

    Description

      Following up on https://issues.redhat.com/browse/MON-1320, we added three new CLI flags to Prometheus to apply different limits on the samples' labels. These new flags are available starting from Prometheus v2.27.0, which will most likely be shipped in OpenShift 4.9.

      The limits that we want to look into for OCP are the following ones:

      # Per-scrape limit on number of labels that will be accepted for a sample. If
# more than this number of labels are present post metric-relabeling, the
# entire scrape will be treated as failed. 0 means no limit.
[ label_limit: <int> | default = 0 ]

# Per-scrape limit on length of labels name that will be accepted for a sample.
# If a label name is longer than this number post metric-relabeling, the entire
# scrape will be treated as failed. 0 means no limit.
[ label_name_length_limit: <int> | default = 0 ]

# Per-scrape limit on length of labels value that will be accepted for a sample.
# If a label value is longer than this number post metric-relabeling, the
# entire scrape will be treated as failed. 0 means no limit.
[ label_value_length_limit: <int> | default = 0 ]

      We could benefit from them by setting relatively high values that could only induce unbound cardinality and thus reject the targets completely if they happened to breach our constrainst.

      DoD:

      • Being able to configure label scrape limits for UWM

      Attachments

        Issue Links

          is blocked by

          Task - A task that needs to be done. MON-1887 Update dependencies to pull prometheus-operator 0.50 changes in CMO

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed

          Task - A task that needs to be done. MON-1653 [upstream] support label scrape limits in Prometheus operator

          • Undefined - Priority not specified.
          • Closed
          is documented by

          Task - A task that needs to be done. RHDEVDOCS-3956 Enforce label scrape limits for UWM

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MON-2258 Label limits not working as expected in prometheus

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          links to

          Web Link Bug Created for node team

          Web Link kubernetes issue for removing metric with large label value

          GitHub openshift/cluster-monitoring-operator#1350: MON-1708: Enforce label scrape limits

          Web Link QE testing pre-merge bug

          Activity

            People

              janantha@redhat.com Jayapriya Pai
              dgrisonn@redhat.com Damien Grisonnet
              Junqi Zhao Junqi Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: