Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-4702

[Upstream] Tempo query frontend failing with error unknown flag: --prometheus.query.support-spanmetrics-connector

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • Tracing Sprint # 259, Tracing Sprint # 260

      Version of components:
      OCP version 4.17.0-0.nightly-2024-09-08-135628
      tempo-operator.v0.13.0

      Description of issue:
      When monitorTab is enabled in the TempoStack instance, the jaeger-query container in the Tempo query frontend pod fails with 

      Error: unknown flag: --prometheus.query.support-spanmetrics-connector

      Steps to reproduce the issue:
      1. Install Tempo Operator built off the latest upstream branch.

      2. Install the OpenTelemetry collector.

      3. Run the red-metrics test case with --skip-delete.

      chainsaw test --config .chainsaw-openshift.yaml  --skip-delete tests/e2e-openshift/red-metrics

      4. The test will fail. Check the TempoStack instance pod status.

      % oc get pods
      NAME                                               READY   STATUS             RESTARTS        AGE
      minio-66774b6457-zj9jz                             1/1     Running            0               11m
      otel-collector-6998f54cfc-phj8s                    1/1     Running            0               11m
      tempo-redmetrics-compactor-656585b745-mqwfr        1/1     Running            0               10m
      tempo-redmetrics-distributor-7c8dd5df5-wdp6s       1/1     Running            0               10m
      tempo-redmetrics-ingester-0                        1/1     Running            0               10m
      tempo-redmetrics-querier-77fd8fcc98-m5j7d          1/1     Running            0               10m
      tempo-redmetrics-query-frontend-79f6794557-fwqrp   3/4     CrashLoopBackOff   6 (4m59s ago)   10m
      

      5. Check the failing pod logs.

      % oc logs tempo-redmetrics-query-frontend-79f6794557-fwqrp -c jaeger-query
      2024/09/09 07:48:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
      WARNING: found deprecated command line option --span-storage.type=grpc, please use environment variable SPAN_STORAGE_TYPE instead
      2024/09/09 07:48:35 application version: git-commit=1da98bb507ed2eee2068dba6c0640c3c5a4f2a50, git-version=v1.60.0, build-date=2024-09-04T03:15:01Z
      Error: unknown flag: --prometheus.query.support-spanmetrics-connector
      unknown flag: --prometheus.query.support-spanmetrics-connector
      Usage:
        jaeger-query [flags]
        jaeger-query [command]
       
      Available Commands:
        completion   Generate the autocompletion script for the specified shell
        docs         Generates documentation
        env          Help about environment variables.
        help         Help about any command
        print-config Print names and values of configuration options
        status       Print the status.
        version      Print the version.
       
      Flags:
            --admin.http.host-port string                The host:port (e.g. 127.0.0.1:16687 or :16687) for the admin server, including health check, /metrics, etc. (default ":16687")
            --admin.http.tls.cert string                 Path to a TLS Certificate file, used to identify this server to clients
            --admin.http.tls.cipher-suites string        Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
            --admin.http.tls.client-ca string            Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
            --admin.http.tls.enabled                     Enable TLS on the server
            --admin.http.tls.key string                  Path to a TLS Private Key file, used to identify this server to clients
            --admin.http.tls.max-version string          Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
            --admin.http.tls.min-version string          Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
            --config-file string                         Configuration file in JSON, TOML, YAML, HCL, or Java properties formats (default none). See spf13/viper for precedence.
            --grpc-storage.connection-timeout duration   The remote storage gRPC server connection timeout (default 5s)
            --grpc-storage.server string                 The remote storage gRPC server address as host:port (default "")
            --grpc-storage.tls.ca string                 Path to a TLS CA (Certification Authority) file used to verify the remote server(s) (by default will use the system truststore)
            --grpc-storage.tls.cert string               Path to a TLS Certificate file, used to identify this process to the remote server(s)
            --grpc-storage.tls.enabled                   Enable TLS when talking to the remote server(s)
            --grpc-storage.tls.key string                Path to a TLS Private Key file, used to identify this process to the remote server(s)
            --grpc-storage.tls.server-name string        Override the TLS server name we expect in the certificate of the remote server(s)
            --grpc-storage.tls.skip-host-verify          (insecure) Skip server's certificate chain and host name verification
        -h, --help                                       help for jaeger-query
            --log-encoding string                        Log encoding. Supported values are 'json' and 'console'. (default "json")
            --log-level string                           Minimal allowed log Level. For more levels see https://github.com/uber-go/zap (default "info")
            --metrics-backend string                     Defines which metrics backend to use for metrics reporting: prometheus or none (default "prometheus")
            --metrics-http-route string                  Defines the route of HTTP endpoint for metrics backends that support scraping (default "/metrics")
            --multi-tenancy.enabled                      Enable tenancy header when receiving or querying
            --multi-tenancy.header string                HTTP header carrying tenant (default "x-tenant")
            --multi-tenancy.tenants string               comma-separated list of allowed values for --multi-tenancy.header header.  (If not supplied, tenants are not restricted)
            --prometheus.connect-timeout duration        The period to wait for a connection to Prometheus when executing queries. (default 30s)
            --prometheus.query.duration-unit string      The units used for the "latency" histogram. It can be either "ms" or "s" and should be consistent with the histogram unit value set in the spanmetrics connector (see: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector#configurations). This also helps jaeger-query determine the metric name when querying for "latency" metrics. (default "ms")
            --prometheus.query.namespace string          The metric namespace that is prefixed to the metric name. A '.' separator will be added between the namespace and the metric name.
            --prometheus.query.normalize-calls           Whether to normalize the "calls" metric name according to https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/translator/prometheus/README.md. For example: "calls" (not normalized) -> "calls_total" (normalized),
            --prometheus.query.normalize-duration        Whether to normalize the "duration" metric name according to https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/translator/prometheus/README.md. For example: "duration_bucket" (not normalized) -> "duration_milliseconds_bucket (normalized)"
            --prometheus.server-url string               The Prometheus server's URL, must include the protocol scheme e.g. http://localhost:9090 (default "http://localhost:9090")
            --prometheus.tls.ca string                   Path to a TLS CA (Certification Authority) file used to verify the remote server(s) (by default will use the system truststore)
            --prometheus.tls.cert string                 Path to a TLS Certificate file, used to identify this process to the remote server(s)
            --prometheus.tls.enabled                     Enable TLS when talking to the remote server(s)
            --prometheus.tls.key string                  Path to a TLS Private Key file, used to identify this process to the remote server(s)
            --prometheus.tls.server-name string          Override the TLS server name we expect in the certificate of the remote server(s)
            --prometheus.tls.skip-host-verify            (insecure) Skip server's certificate chain and host name verification
            --prometheus.token-file string               The path to a file containing the bearer token which will be included when executing queries against the Prometheus API.
            --prometheus.token-override-from-context     Whether the bearer token should be overridden from context (incoming request) (default true)
            --query.additional-headers strings           Additional HTTP response headers.  Can be specified multiple times.  Format: "Key: Value" (default [])
            --query.base-path string                     The base path for all HTTP routes, e.g. /jaeger; useful when running behind a reverse proxy. See https://github.com/jaegertracing/jaeger/blob/main/examples/reverse-proxy/README.md (default "/")
            --query.bearer-token-propagation             Allow propagation of bearer token to be used by storage plugins (default false)
            --query.enable-tracing                       Enables emitting jaeger-query traces
            --query.grpc-server.host-port string         The host:port (e.g. 127.0.0.1:14250 or :14250) of the query's gRPC server (default ":16685")
            --query.grpc.tls.cert string                 Path to a TLS Certificate file, used to identify this server to clients
            --query.grpc.tls.cipher-suites string        Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
            --query.grpc.tls.client-ca string            Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
            --query.grpc.tls.enabled                     Enable TLS on the server
            --query.grpc.tls.key string                  Path to a TLS Private Key file, used to identify this server to clients
            --query.grpc.tls.max-version string          Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
            --query.grpc.tls.min-version string          Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
            --query.http-server.host-port string         The host:port (e.g. 127.0.0.1:14268 or :14268) of the query's HTTP server (default ":16686")
            --query.http.tls.cert string                 Path to a TLS Certificate file, used to identify this server to clients
            --query.http.tls.cipher-suites string        Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
            --query.http.tls.client-ca string            Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
            --query.http.tls.enabled                     Enable TLS on the server
            --query.http.tls.key string                  Path to a TLS Private Key file, used to identify this server to clients
            --query.http.tls.max-version string          Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
            --query.http.tls.min-version string          Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
            --query.log-static-assets-access             Log when static assets are accessed (for debugging)
            --query.max-clock-skew-adjustment duration   The maximum delta by which span timestamps may be adjusted in the UI due to clock skew; set to 0s to disable clock skew adjustments (default 0s)
            --query.static-files string                  The directory path override for the static assets for the UI
            --query.ui-config string                     The path to the UI configuration file in JSON format
            --span-storage.type string                   (deprecated) please use SPAN_STORAGE_TYPE environment variable. Run this binary with the 'env' command for help. (default "")
       
      Use "jaeger-query [command] --help" for more information about a command.
      

       

              bbongart@redhat.com Benedikt Bongartz
              rhn-support-ikanse Ishwar Kanse
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: