Details
-
Bug
-
Resolution: Obsolete
-
Critical
-
None
-
None
-
None
-
None
-
Kiali Sprint #27
Description
Test Source: https://issues.jboss.org/browse/TRACING-582
- redundancyPolicy: FullRedundancy
- nodeCount: 1
Images/Versions:
Componenet | Image/Version |
---|---|
ES Operator | quay.io/openshift/origin-elasticsearch-operator:4.1 |
ES Operator proxy | quay.io/openshift/origin-oauth-proxy:v4.0.0 |
Jaeger Operator | brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/distributed-tracing-tech-preview/jaeger-operator:1.13.1-3 |
Jaeger agent | brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/distributed-tracing-tech-preview/jaeger-agent:1.13.1-1 |
Jaeger collector | brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/distributed-tracing-tech-preview/jaeger-collector:1.13.1-1 |
Jaeger query | brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/distributed-tracing-tech-preview/jaeger-query:1.13.1-1 |
Elasticsearch image | quay.io/openshift/origin-logging-elasticsearch5:4.1 |
OpenShift | OCP 4.1.4 |
Kubernetes Master | v1.13.4+c62ce01 |
Setup: (CR file attached, test_redty_full_1.yaml)
- Deploy jaeger services with the given CR file
- Once all services up, go to jaeger UI page and click find traces (few traces will be created)
- on the command line delete the ES pod (oc delete pod <es-pod-name>)
- ES pod/node will be redeployed and will be available in few minutes
- From this point UI always return ES API 400 error (jaeger-query logs as follows)
- I see jaeger-collector works as expected
UI error:
There was an error querying for traces: HTTP Error: Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception] Status 500 Status text Internal Server Error URL /api/traces Query end=1564129421601000&limit=20&lookback=1h&maxDuration&minDuration&service=jaeger-query&start=1564125821601000 Response body { "data": null, "total": 0, "limit": 0, "offset": 0, "errors": [ { "code": 500, "msg": "Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]" } ] }
jaeger-query log:
2019/07/26 08:18:19 maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS {"level":"info","ts":1564129099.2232416,"caller":"flags/service.go:115","msg":"Mounting metrics handler on admin server","route":"/metrics"} {"level":"info","ts":1564129099.2235725,"caller":"flags/admin.go:108","msg":"Mounting health check on admin server","route":"/"} {"level":"info","ts":1564129099.2237003,"caller":"flags/admin.go:114","msg":"Starting admin HTTP server","http-port":16687} {"level":"info","ts":1564129099.2237113,"caller":"flags/admin.go:100","msg":"Admin server started","http-port":16687,"health-status":"unavailable"} {"level":"info","ts":1564129099.2777426,"caller":"healthcheck/handler.go:129","msg":"Health Check state change","status":"ready"} {"level":"info","ts":1564129099.2777987,"caller":"app/server.go:134","msg":"Starting CMUX server","port":16686} {"level":"info","ts":1564129099.2778442,"caller":"app/server.go:111","msg":"Starting HTTP server","port":16686} {"level":"info","ts":1564129099.2778673,"caller":"app/server.go:124","msg":"Starting GRPC server","port":16686} {"level":"error","ts":1564129244.8093076,"caller":"app/http_handler.go:409","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","errorVerbose":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).findTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:487\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:294\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:275\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:78\ngithub.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:88\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:202\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:409\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:203\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847"} {"level":"error","ts":1564129258.3651261,"caller":"app/http_handler.go:409","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","errorVerbose":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).findTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:487\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:294\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:275\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:78\ngithub.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:88\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:202\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:409\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:203\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847"} {"level":"error","ts":1564129259.2842135,"caller":"app/http_handler.go:409","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","errorVerbose":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).findTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:487\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:294\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:275\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:78\ngithub.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:88\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:202\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:409\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:203\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847"} {"level":"error","ts":1564129260.1025317,"caller":"app/http_handler.go:409","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","errorVerbose":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).findTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:487\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:294\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:275\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:78\ngithub.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:88\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:202\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:409\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:203\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847"} {"level":"error","ts":1564129260.9062722,"caller":"app/http_handler.go:409","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","errorVerbose":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).findTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:487\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:294\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:275\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:78\ngithub.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:88\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:202\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:409\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:203\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847"} {"level":"error","ts":1564129354.5627081,"caller":"app/http_handler.go:409","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","errorVerbose":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).findTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:487\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraceIDs\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:294\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:275\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:78\ngithub.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.FindTraces\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:88\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:202\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:409\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:203\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:132\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:135\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:212\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat1/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/net/http/server.go:1847"}
Attachments
Issue Links
- relates to
-
TRACING-629 Add index mappings to ECL image
- Closed