-
Bug
-
Resolution: Done
-
Normal
-
None
-
ACM 2.8.0
-
1
-
False
-
None
-
False
-
-
-
Observability Sprint 2023-11, Observability Sprint 2023-15
-
Moderate
-
No
Description of problem: there are some errors generated each hour in the receive pod, but metrics data are displaying on the Grafana.
Version-Release number of selected component (if applicable): 2.8.0-155
How reproducible:
Steps to Reproduce:
- Deploy MCOCR with default YAML
- all the pods are running
- in the receive pods, there are some errors are generated in each hour
```
level=warn ts=2023-05-06T08:36:58.914704349Z caller=shipper.go:239 component=receive component=multi-tsdb tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="reading meta file failed, will override it" err="failed to read /var/thanos/receive/28e07f24-8716-4a5c-bc49-818979b5c18c/thanos.shipper.json: open /var/thanos/receive/28e07f24-8716-4a5c-bc49-818979b5c18c/thanos.shipper.json: no such file or directory"
level=warn ts=2023-05-06T09:53:57.923217492Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=36
level=warn ts=2023-05-06T09:53:57.923277065Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=48
level=warn ts=2023-05-06T09:53:57.923990085Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=37
level=warn ts=2023-05-06T09:53:57.924045306Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=44
level=warn ts=2023-05-06T09:53:57.925783272Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=36
level=warn ts=2023-05-06T09:53:57.925844501Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=54
level=warn ts=2023-05-06T10:34:52.647745366Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=228
level=warn ts=2023-05-06T10:34:52.647811486Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=25
level=warn ts=2023-05-06T10:34:52.64784422Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=199
level=warn ts=2023-05-06T10:34:52.647896445Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=45
level=warn ts=2023-05-06T10:34:52.649153024Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=202
level=warn ts=2023-05-06T10:34:52.649230102Z caller=writer.go:210 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting samples with different value but same timestamp" numDropped=37
level=warn ts=2023-05-06T10:41:48.381567628Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" nu
mDropped=539
level=warn ts=2023-05-06T10:41:48.38196581Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=586
level=warn ts=2023-05-06T10:41:48.382553561Z caller=writer.go:206 component=receive component=receive-writer tenant=28e07f24-8716-4a5c-bc49-818979b5c18c msg="Error on ingesting out-of-order samples" numDropped=549
```
Actual results:
Expected results:
Additional info:
- account is impacted by
-
ACM-5291 [ACM QE] Observability - ACM 2.8 Train 04 (Global Sprint 2023 - 05) QE Work Items
- Closed