Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-5373

Thanos-receive is dropping out-of-order samples in 2.8.2 MCO

XMLWordPrintable

    • 1
    • False
    • None
    • False
    • Observability Sprint 2023-11, Observability Sprint 2023-15
    • Moderate
    • No

      Description of problem: there are some errors generated each hour in the receive pod, but metrics data are displaying on the Grafana.

      Version-Release number of selected component (if applicable): 2.8.0-155

      How reproducible:

      Steps to Reproduce:

      1. Deploy MCOCR with default YAML
      2. all the pods are running
      3. in the receive pods, there are some errors are generated in each hour
        ```
        level=warn ts=2023-05-06T08:36:58.914704349Z caller=shipper.go:239 component=receive component=multi-tsdb tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="reading meta file failed, will override it" err="failed to read /var/thanos/receive/28e07f24-8716-4a5c-bc49-818979b5c18c/thanos.shipper.json: open /var/thanos/receive/28e07f24-8716-4a5c-bc49-818979b5c18c/thanos.shipper.json: no such file or directory"
        level=warn ts=2023-05-06T09:53:57.923217492Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=36
        level=warn ts=2023-05-06T09:53:57.923277065Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=48
        level=warn ts=2023-05-06T09:53:57.923990085Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=37
        level=warn ts=2023-05-06T09:53:57.924045306Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=44
        level=warn ts=2023-05-06T09:53:57.925783272Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=36
        level=warn ts=2023-05-06T09:53:57.925844501Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=54
        level=warn ts=2023-05-06T10:34:52.647745366Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=228
        level=warn ts=2023-05-06T10:34:52.647811486Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=25
        level=warn ts=2023-05-06T10:34:52.64784422Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=199
        level=warn ts=2023-05-06T10:34:52.647896445Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=45
        level=warn ts=2023-05-06T10:34:52.649153024Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=202
        level=warn ts=2023-05-06T10:34:52.649230102Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=37
        level=warn ts=2023-05-06T10:41:48.381567628Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" nu
        mDropped=539
        level=warn ts=2023-05-06T10:41:48.38196581Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=586
        level=warn ts=2023-05-06T10:41:48.382553561Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=549
        ```

      Actual results:

      Expected results:

      Additional info:

            smeduri1@redhat.com Subbarao Meduri
            cquredhat ChangLiang Qu
            ACM QE Team
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: