Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-38690

Prometheus no longer accepts samples of the same series with different timestamps

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • None
    • 4.16
    • Monitoring
    • None
    • Important
    • Yes
    • MON Sprint 258
    • 1
    • False
    • Hide

      None

      Show
      None
    • (4.17.0 not released yet and we're aiming 4.17.0 with the fix for now)
    • Release Note Not Required
    • Done

      This is a clone of issue OCPBUGS-38622. The following is the description of the original issue:
      —
      Description of problem:

          See https://github.com/prometheus/prometheus/issues/14503 for more details

      Version-Release number of selected component (if applicable): 

          4.16

      How reproducible:

      Steps to Reproduce:
      1. Make Prometheus scrape a target that exposes multiple samples of the same series with different explicit timestamps, for example:

      # TYPE requests_per_second_requests gauge
# UNIT requests_per_second_requests requests
# HELP requests_per_second_requests test-description
requests_per_second_requests 16 1722466225604
requests_per_second_requests 14 1722466226604
requests_per_second_requests 40 1722466227604
requests_per_second_requests 15 1722466228604
# EOF

      2. Not all the samples will be ingested
      3. If Prometheus continues scraping that target for a moment, the PrometheusDuplicateTimestamps will fire.
      Actual results:

      Expected results: all the samples should be considered (or course if the timestamps are too old or are too in the future, Prometheus may refuses them.)

      Additional info: 

           Regression introduced in Prometheus 2.52.
    Proposed upstream fixes: https://github.com/prometheus/prometheus/pull/14683 https://github.com/prometheus/prometheus/pull/14685

            rh-ee-amrini Ayoub Mrini
            openshift-crt-jira-prow OpenShift Prow Bot
            Junqi Zhao Junqi Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: