Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-39179

Prometheus no longer accepts samples of the same series with different timestamps

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • 4.16.z
    • 4.16
    • Monitoring
    • Important
    • Yes
    • MON Sprint 258, MON Sprint 259
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, during the same scrape, Prometheus would drop samples from the same series and only consider one of them, even when they had different timestamps. When this issue occurred continuously, it triggered the `PrometheusDuplicateTimestamps` alert. With this release, all samples are now ingested if they meet the other conditions. (link:https://issues.redhat.com/browse/OCPBUGS-39179[*OCPBUGS-39179*])
      Show
      * Previously, during the same scrape, Prometheus would drop samples from the same series and only consider one of them, even when they had different timestamps. When this issue occurred continuously, it triggered the `PrometheusDuplicateTimestamps` alert. With this release, all samples are now ingested if they meet the other conditions. (link: https://issues.redhat.com/browse/OCPBUGS-39179 [* OCPBUGS-39179 *])
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-38690. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-38622. The following is the description of the original issue:

      Description of problem:

          See https://github.com/prometheus/prometheus/issues/14503 for more details
          

      Version-Release number of selected component (if applicable):

          4.16
          

      How reproducible:

      
          

      Steps to Reproduce:
      1. Make Prometheus scrape a target that exposes multiple samples of the same series with different explicit timestamps, for example:

      # TYPE requests_per_second_requests gauge
      # UNIT requests_per_second_requests requests
      # HELP requests_per_second_requests test-description
      requests_per_second_requests 16 1722466225604
      requests_per_second_requests 14 1722466226604
      requests_per_second_requests 40 1722466227604
      requests_per_second_requests 15 1722466228604
      # EOF
      

      2. Not all the samples will be ingested
      3. If Prometheus continues scraping that target for a moment, the PrometheusDuplicateTimestamps will fire.
      Actual results:

      
          

      Expected results: all the samples should be considered (or course if the timestamps are too old or are too in the future, Prometheus may refuses them.)

      
          

      Additional info:

           Regression introduced in Prometheus 2.52.
          Proposed upstream fixes: https://github.com/prometheus/prometheus/pull/14683 https://github.com/prometheus/prometheus/pull/14685 
          

              rh-ee-amrini Ayoub Mrini
              openshift-crt-jira-prow OpenShift Prow Bot
              Junqi Zhao Junqi Zhao
              Eliska Romanova Eliska Romanova
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: