-
Bug
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
Version of components:
OCP version 4.17.0-0.nightly-2024-09-08-135628
tempo-operator.v0.13.0
Description of issue:
When monitorTab is enabled in the TempoStack instance, the jaeger-query container in the Tempo query frontend pod fails with
Error: unknown flag: --prometheus.query.support-spanmetrics-connector
Steps to reproduce the issue:
1. Install Tempo Operator built off the latest upstream branch.
2. Install the OpenTelemetry collector.
3. Run the red-metrics test case with --skip-delete.
chainsaw test --config .chainsaw-openshift.yaml --skip-delete tests/e2e-openshift/red-metrics
4. The test will fail. Check the TempoStack instance pod status.
% oc get pods NAME READY STATUS RESTARTS AGE minio-66774b6457-zj9jz 1/1 Running 0 11m otel-collector-6998f54cfc-phj8s 1/1 Running 0 11m tempo-redmetrics-compactor-656585b745-mqwfr 1/1 Running 0 10m tempo-redmetrics-distributor-7c8dd5df5-wdp6s 1/1 Running 0 10m tempo-redmetrics-ingester-0 1/1 Running 0 10m tempo-redmetrics-querier-77fd8fcc98-m5j7d 1/1 Running 0 10m tempo-redmetrics-query-frontend-79f6794557-fwqrp 3/4 CrashLoopBackOff 6 (4m59s ago) 10m
5. Check the failing pod logs.
% oc logs tempo-redmetrics-query-frontend-79f6794557-fwqrp -c jaeger-query 2024/09/09 07:48:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined WARNING: found deprecated command line option --span-storage.type=grpc, please use environment variable SPAN_STORAGE_TYPE instead 2024/09/09 07:48:35 application version: git-commit=1da98bb507ed2eee2068dba6c0640c3c5a4f2a50, git-version=v1.60.0, build-date=2024-09-04T03:15:01Z Error: unknown flag: --prometheus.query.support-spanmetrics-connector unknown flag: --prometheus.query.support-spanmetrics-connector Usage: jaeger-query [flags] jaeger-query [command] Available Commands: completion Generate the autocompletion script for the specified shell docs Generates documentation env Help about environment variables. help Help about any command print-config Print names and values of configuration options status Print the status. version Print the version. Flags: --admin.http.host-port string The host:port (e.g. 127.0.0.1:16687 or :16687) for the admin server, including health check, /metrics, etc. (default ":16687") --admin.http.tls.cert string Path to a TLS Certificate file, used to identify this server to clients --admin.http.tls.cipher-suites string Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). --admin.http.tls.client-ca string Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted) --admin.http.tls.enabled Enable TLS on the server --admin.http.tls.key string Path to a TLS Private Key file, used to identify this server to clients --admin.http.tls.max-version string Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3) --admin.http.tls.min-version string Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3) --config-file string Configuration file in JSON, TOML, YAML, HCL, or Java properties formats (default none). See spf13/viper for precedence. --grpc-storage.connection-timeout duration The remote storage gRPC server connection timeout (default 5s) --grpc-storage.server string The remote storage gRPC server address as host:port (default "") --grpc-storage.tls.ca string Path to a TLS CA (Certification Authority) file used to verify the remote server(s) (by default will use the system truststore) --grpc-storage.tls.cert string Path to a TLS Certificate file, used to identify this process to the remote server(s) --grpc-storage.tls.enabled Enable TLS when talking to the remote server(s) --grpc-storage.tls.key string Path to a TLS Private Key file, used to identify this process to the remote server(s) --grpc-storage.tls.server-name string Override the TLS server name we expect in the certificate of the remote server(s) --grpc-storage.tls.skip-host-verify (insecure) Skip server's certificate chain and host name verification -h, --help help for jaeger-query --log-encoding string Log encoding. Supported values are 'json' and 'console'. (default "json") --log-level string Minimal allowed log Level. For more levels see https://github.com/uber-go/zap (default "info") --metrics-backend string Defines which metrics backend to use for metrics reporting: prometheus or none (default "prometheus") --metrics-http-route string Defines the route of HTTP endpoint for metrics backends that support scraping (default "/metrics") --multi-tenancy.enabled Enable tenancy header when receiving or querying --multi-tenancy.header string HTTP header carrying tenant (default "x-tenant") --multi-tenancy.tenants string comma-separated list of allowed values for --multi-tenancy.header header. (If not supplied, tenants are not restricted) --prometheus.connect-timeout duration The period to wait for a connection to Prometheus when executing queries. (default 30s) --prometheus.query.duration-unit string The units used for the "latency" histogram. It can be either "ms" or "s" and should be consistent with the histogram unit value set in the spanmetrics connector (see: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector#configurations). This also helps jaeger-query determine the metric name when querying for "latency" metrics. (default "ms") --prometheus.query.namespace string The metric namespace that is prefixed to the metric name. A '.' separator will be added between the namespace and the metric name. --prometheus.query.normalize-calls Whether to normalize the "calls" metric name according to https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/translator/prometheus/README.md. For example: "calls" (not normalized) -> "calls_total" (normalized), --prometheus.query.normalize-duration Whether to normalize the "duration" metric name according to https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/translator/prometheus/README.md. For example: "duration_bucket" (not normalized) -> "duration_milliseconds_bucket (normalized)" --prometheus.server-url string The Prometheus server's URL, must include the protocol scheme e.g. http://localhost:9090 (default "http://localhost:9090") --prometheus.tls.ca string Path to a TLS CA (Certification Authority) file used to verify the remote server(s) (by default will use the system truststore) --prometheus.tls.cert string Path to a TLS Certificate file, used to identify this process to the remote server(s) --prometheus.tls.enabled Enable TLS when talking to the remote server(s) --prometheus.tls.key string Path to a TLS Private Key file, used to identify this process to the remote server(s) --prometheus.tls.server-name string Override the TLS server name we expect in the certificate of the remote server(s) --prometheus.tls.skip-host-verify (insecure) Skip server's certificate chain and host name verification --prometheus.token-file string The path to a file containing the bearer token which will be included when executing queries against the Prometheus API. --prometheus.token-override-from-context Whether the bearer token should be overridden from context (incoming request) (default true) --query.additional-headers strings Additional HTTP response headers. Can be specified multiple times. Format: "Key: Value" (default []) --query.base-path string The base path for all HTTP routes, e.g. /jaeger; useful when running behind a reverse proxy. See https://github.com/jaegertracing/jaeger/blob/main/examples/reverse-proxy/README.md (default "/") --query.bearer-token-propagation Allow propagation of bearer token to be used by storage plugins (default false) --query.enable-tracing Enables emitting jaeger-query traces --query.grpc-server.host-port string The host:port (e.g. 127.0.0.1:14250 or :14250) of the query's gRPC server (default ":16685") --query.grpc.tls.cert string Path to a TLS Certificate file, used to identify this server to clients --query.grpc.tls.cipher-suites string Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). --query.grpc.tls.client-ca string Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted) --query.grpc.tls.enabled Enable TLS on the server --query.grpc.tls.key string Path to a TLS Private Key file, used to identify this server to clients --query.grpc.tls.max-version string Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3) --query.grpc.tls.min-version string Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3) --query.http-server.host-port string The host:port (e.g. 127.0.0.1:14268 or :14268) of the query's HTTP server (default ":16686") --query.http.tls.cert string Path to a TLS Certificate file, used to identify this server to clients --query.http.tls.cipher-suites string Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). --query.http.tls.client-ca string Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted) --query.http.tls.enabled Enable TLS on the server --query.http.tls.key string Path to a TLS Private Key file, used to identify this server to clients --query.http.tls.max-version string Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3) --query.http.tls.min-version string Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3) --query.log-static-assets-access Log when static assets are accessed (for debugging) --query.max-clock-skew-adjustment duration The maximum delta by which span timestamps may be adjusted in the UI due to clock skew; set to 0s to disable clock skew adjustments (default 0s) --query.static-files string The directory path override for the static assets for the UI --query.ui-config string The path to the UI configuration file in JSON format --span-storage.type string (deprecated) please use SPAN_STORAGE_TYPE environment variable. Run this binary with the 'env' command for help. (default "") Use "jaeger-query [command] --help" for more information about a command.