Uploaded image for project: 'Observability and Data Analysis Program'
  1. Observability and Data Analysis Program
  2. OBSDA-205

[FEATURE] allow to configure enforced limits on PrometheusK8sConfig

XMLWordPrintable

    • Icon: Feature Feature
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • None
    • PM Monitoring
    • None
    • False
    • None
    • False
    • Not Selected

      Hello,

      I would like to add theses enforced limits on ClusterMonitoringConfiguration and not only for UserWorkloadConfiguration:

      • enforcedLabelLimit
      • enforcedLabelNameLengthLimit
      • enforcedLabelValueLengthLimit
      • enforcedSampleLimit
      • enforcedTargetLimit

      According to the doc,

      we can only setup these limit for UserWorkloadConfiguration.

      I have the following use case:

      We have several openshift clusters which push all metrics by the remote write to our grafana mimir cluster.

      We have the Compliance operator which produce the metric compliance_operator_compliance_scan_error_total but this metric has the label error value which take more than 11K characters. This is not acceptable and this is blocked by our mimir cluster:

      The default mimir max-length-label-value is 2048.

      rpc error: code = Code(400) desc = received a series whose label value length exceeds the limit, value: 'TitlertConfigure A Unique CA Certificate for etcdnRulertxccdf_org.ssgproject.content_rule_etcd_unique_canResultrtpassnnTitlertDisable Anonymous Authentication to the KubeletnRulertxccdf_org.ssgproject' (truncated) series: 'compliance_operator_compliance_scan_error_total{cluster="c01e1", endpoint="metrics-co", error="Title\r\tConfigure A Unique CA Certificate for etcd\nRule\r\txccdf_org.ssgproject.content_rule_etcd_uniqu' (err-mimir-label-value-too-long). To adjust the related per-tenant limit, configure -validation.max-length-label-value, or contact your service administrator. 

       

      So, in order to reduce severals log message from mimir, I want to add the enforce limit to prevent to scrape this not acceptable metric.

              rh-ee-rfloren Roger Florén
              francois.gouteroux@gmail.com François GOUTEROUX (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: