Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: rhosdt-3.8
Affects Version/s: None
Component/s: None
Labels:
None

Activity Type:
Quality / Stability / Reliability
Story Points:
1
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Git Pull Request:
https://github.com/grafana/tempo-operator/pull/1274
Intelligence Requested:
Market:

Sprint:
Tracing Sprint # 275, Tracing Sprint # 276, Tracing Sprint # 277 - 3.7, Tracing Sprint # 278

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of the problem:

When metrics collection is enabled on a Tempo Monolithc instance with multitenancy enabled, the metrics exposed doesn't contain metrics from all the Tempo components.

Steps to reproduce the issue:

Fetch the PR commit https://github.com/grafana/tempo-operator/pull/1230
Run the monolithic-multitenancy-openshift test with --skip-delete.

chainsaw test --skip-delete --config .chainsaw-openshift.yaml tests/e2e-openshift/monolithic-multitenancy-openshift

Port forward the metrics port from svc.

oc port-forward svc/tempo-mmo-gateway 8081:8081

Curl the metrics and check that component metrics are missing.

% curl http://localhost:8081/metrics                 
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.2245e-05
go_gc_duration_seconds{quantile="0.25"} 4.4637e-05
go_gc_duration_seconds{quantile="0.5"} 5.5961e-05
go_gc_duration_seconds{quantile="0.75"} 7.8946e-05
go_gc_duration_seconds{quantile="1"} 0.000183549
go_gc_duration_seconds_sum 0.000522698
go_gc_duration_seconds_count 7
# HELP go_gc_gogc_percent Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. Sourced from /gc/gogc:percent
# TYPE go_gc_gogc_percent gauge
go_gc_gogc_percent 100
# HELP go_gc_gomemlimit_bytes Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. Sourced from /gc/gomemlimit:bytes
# TYPE go_gc_gomemlimit_bytes gauge
go_gc_gomemlimit_bytes 9.223372036854776e+18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 36
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.23.9 (Red Hat 1.23.9-1.module+el8.10.0+23162+9223a61a) X:strictfipsruntime"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated in heap and currently in use. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.1934096e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated in heap until now, even if released already. Equals to /gc/heap/allocs:bytes.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 2.1220624e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. Equals to /memory/classes/profiling/buckets:bytes.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.458811e+06
# HELP go_memstats_frees_total Total number of heap objects frees. Equals to /gc/heap/frees:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 73137
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. Equals to /memory/classes/metadata/other:bytes.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.738752e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and currently in use, same as go_memstats_alloc_bytes. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.1934096e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. Equals to /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.390912e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.540096e+07
# HELP go_memstats_heap_objects Number of currently allocated objects. Equals to /gc/heap/objects:objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 60847
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. Equals to /memory/classes/heap/released:bytes.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.333568e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes + /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.9791872e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7507490811577451e+09
# HELP go_memstats_mallocs_total Total number of heap objects allocated, both live and gc-ed. Semantically a counter version for go_memstats_heap_objects gauge. Equals to /gc/heap/allocs:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 133984
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. Equals to /memory/classes/metadata/mcache/inuse:bytes.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. Equals to /memory/classes/metadata/mcache/inuse:bytes + /memory/classes/metadata/mcache/free:bytes.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. Equals to /memory/classes/metadata/mspan/inuse:bytes.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 198400
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. Equals to /memory/classes/metadata/mspan/inuse:bytes + /memory/classes/metadata/mspan/free:bytes.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 212160
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. Equals to /gc/heap/goal:bytes.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.5346384e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. Equals to /memory/classes/other:bytes.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 822365
# HELP go_memstats_stack_inuse_bytes Number of bytes obtained from system for stack allocator in non-CGO environments. Equals to /memory/classes/heap/stacks:bytes.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.048576e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. Equals to /memory/classes/heap/stacks:bytes + /memory/classes/os-stacks:bytes.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.048576e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system. Equals to /memory/classes/total:byte.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.7088136e+07
# HELP go_sched_gomaxprocs_threads The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously. Sourced from /sched/gomaxprocs:threads
# TYPE go_sched_gomaxprocs_threads gauge
go_sched_gomaxprocs_threads 4
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 10
# HELP healthcheck Indicates if check is healthy (1 is healthy, 0 is unhealthy)
# TYPE healthcheck gauge
healthcheck{check="live",name="http"} 1
# HELP http_proxy_requests_total Counter of proxy HTTP requests.
# TYPE http_proxy_requests_total counter
http_proxy_requests_total{method="GET",proxy="tracesv1-read"} 4
http_proxy_requests_total{method="POST",proxy="tracesv1-read"} 1
# HELP http_request_duration_seconds Histogram of latencies for HTTP requests.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricslegacy",handler="query",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricslegacy",handler="query",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricslegacy",handler="query_range",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricslegacy",handler="query_range",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="alerts",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="alerts",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="labels",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="labels",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="labelvalues",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="labelvalues",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="query",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="query",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="query_range",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="query_range",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="receive",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="receive",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="rules",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="rules",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="rules-raw",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="rules-raw",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="series",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="series",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="silences",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="silences",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="0.1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="0.2"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="0.4"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="1"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="2.5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="5"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="8"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="20"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="60"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="120"} 0
http_request_duration_seconds_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="+Inf"} 0
http_request_duration_seconds_sum{code="",group="metricsv1",handler="ui",method="",tenant=""} 0
http_request_duration_seconds_count{code="",group="metricsv1",handler="ui",method="",tenant=""} 0
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="0.1"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="0.2"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="0.4"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="2.5"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="5"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="8"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="20"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="60"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="120"} 1
http_request_duration_seconds_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="+Inf"} 1
http_request_duration_seconds_sum{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 0.001604516
http_request_duration_seconds_count{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 1
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="0.1"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="0.2"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="0.4"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="2.5"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="5"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="8"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="20"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="60"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="120"} 4
http_request_duration_seconds_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="+Inf"} 4
http_request_duration_seconds_sum{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 0.082948726
http_request_duration_seconds_count{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 4
# HELP http_request_size_bytes Size of HTTP requests.
# TYPE http_request_size_bytes summary
http_request_size_bytes_sum{code="",group="metricslegacy",handler="query",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricslegacy",handler="query",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricslegacy",handler="query_range",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricslegacy",handler="query_range",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="alerts",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="alerts",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="labels",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="labels",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="labelvalues",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="labelvalues",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="query",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="query",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="query_range",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="query_range",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="receive",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="receive",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="rules",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="rules",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="rules-raw",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="rules-raw",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="series",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="series",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="silences",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="silences",method="",tenant=""} 0
http_request_size_bytes_sum{code="",group="metricsv1",handler="ui",method="",tenant=""} 0
http_request_size_bytes_count{code="",group="metricsv1",handler="ui",method="",tenant=""} 0
http_request_size_bytes_sum{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 1111
http_request_size_bytes_count{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 1
http_request_size_bytes_sum{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 768
http_request_size_bytes_count{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 4
# HELP http_requests_total Counter of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{code="",group="metricslegacy",handler="query",method="",tenant=""} 0
http_requests_total{code="",group="metricslegacy",handler="query_range",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="alerts",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="labels",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="labelvalues",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="query",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="query_range",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="receive",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="rules",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="rules-raw",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="series",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="silences",method="",tenant=""} 0
http_requests_total{code="",group="metricsv1",handler="ui",method="",tenant=""} 0
http_requests_total{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 1
http_requests_total{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 4
# HELP http_response_size_bytes Histogram of response size for HTTP requests.
# TYPE http_response_size_bytes histogram
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricslegacy",handler="query",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricslegacy",handler="query",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricslegacy",handler="query_range",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricslegacy",handler="query_range",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricslegacy",handler="query_range",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="alerts",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="alerts",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="alerts",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labels",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="labels",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="labels",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="labelvalues",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="labelvalues",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="labelvalues",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="query",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="query",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="query_range",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="query_range",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="query_range",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="receive",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="receive",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="receive",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="rules",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="rules",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="rules-raw",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="rules-raw",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="rules-raw",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="series",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="series",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="series",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="silences",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="silences",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="silences",method="",tenant=""} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="100"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="1000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="10000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="100000"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="1e+06"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="1e+07"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="1e+08"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="1e+09"} 0
http_response_size_bytes_bucket{code="",group="metricsv1",handler="ui",method="",tenant="",le="+Inf"} 0
http_response_size_bytes_sum{code="",group="metricsv1",handler="ui",method="",tenant=""} 0
http_response_size_bytes_count{code="",group="metricsv1",handler="ui",method="",tenant=""} 0
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="100"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1000"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="10000"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="100000"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+06"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+07"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+08"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+09"} 1
http_response_size_bytes_bucket{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="+Inf"} 1
http_response_size_bytes_sum{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 2
http_response_size_bytes_count{code="200",group="tracesotlphttpv1api",handler="traces",method="POST",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 1
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="100"} 0
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1000"} 0
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="10000"} 2
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="100000"} 4
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+06"} 4
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+07"} 4
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+08"} 4
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="1e+09"} 4
http_response_size_bytes_bucket{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa",le="+Inf"} 4
http_response_size_bytes_sum{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 41269
http_response_size_bytes_count{code="200",group="tracesv1api",handler="traces",method="GET",tenant="1610b0c3-c509-4592-a256-a1871353dbfa"} 4
# HELP observatorium_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which observatorium was built, and the goos and goarch for the build.
# TYPE observatorium_build_info gauge
observatorium_build_info{branch="",goarch="amd64",goos="linux",goversion="go1.23.9 (Red Hat 1.23.9-1.module+el8.10.0+23162+9223a61a) X:strictfipsruntime",revision="unknown",tags="strictfipsruntime",version=""} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.17
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_network_receive_bytes_total Number of bytes received by the process over the network.
# TYPE process_network_receive_bytes_total counter
process_network_receive_bytes_total 390839
# HELP process_network_transmit_bytes_total Number of bytes sent by the process over the network.
# TYPE process_network_transmit_bytes_total counter
process_network_transmit_bytes_total 447606
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 20
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 6.8120576e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.75074902772e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.94138112e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19

Additional details:

All the component metrics are exposed when using non-multitenant Tempo Monolithc instance. This can be checked by running the monitoring-monolithic test with --skip-delete and port-forwarding to the metrics port.

Assignee:: Ruben Vargas Palma

Reporter:: Ishwar Kanse

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2025/06/24 6:47 AM

Updated:: 2025/12/02 3:17 PM

Resolved:: 2025/10/19 5:43 AM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates