[COO-14] Thanos Querier service can't reach pod

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: 0.1.2
Affects Version/s: None
Component/s: None
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Intelligence Requested:
Market:

Sprint:
MON Sprint 251

Regression:
No

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

The thanos container only listens to 127.0.0.1. If one tries to reach the Service a 502 is returned.

A workaround is to created the deployment and service manually

kind: Deployment
apiVersion: apps/v1
metadata: 
  name: thanos-querier-example-thanos
  namespace: default
  labels: 
    app.kubernetes.io/instance: thanos-querier-example-thanos
    app.kubernetes.io/managed-by: observability-operator
    app.kubernetes.io/part-of: ThanosQuerier
spec: 
  replicas: 1
  selector: 
    matchLabels: 
      app.kubernetes.io/instance: thanos-querier-example-thanos
  template: 
    metadata: 
      name: thanos-querier-example-thanos
      namespace: default
      creationTimestamp: null
      labels: 
        app.kubernetes.io/instance: thanos-querier-example-thanos
        app.kubernetes.io/managed-by: observability-operator
        app.kubernetes.io/part-of: ThanosQuerier
    spec: 
      containers: 
        - name: thanos-querier
          image: 'registry.redhat.io/cluster-observability-operator/coo-thanos-rhel8@sha256:79d1a7f1568800f06a7146e219852a42a3ccd265e626d2d476640f83f2c901bd'
          args: 
            - query
            - '--grpc-address=127.0.0.1:10901'
            - '--http-address=:9090'
            - '--log.format=logfmt'
            - '--query.replica-label=prometheus_replica'
            - '--query.auto-downsampling'
            - '--endpoint=dnssrv+_grpc._tcp.sample-monitoring-stack-thanos-sidecar.default.svc.cluster.local'
          ports: 
            - name: metrics
              containerPort: 9090
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: FallbackToLogsOnError
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      nodeSelector: 
        kubernetes.io/os: linux
      securityContext: {}
      schedulerName: default-scheduler
  strategy: 
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 300
---
kind: Service
apiVersion: v1
metadata: 
  name: thanos-querier
  namespace: default
spec: 
  ports: 
    - name: http
      protocol: TCP
      port: 9090
      targetPort: 9090
  type: ClusterIP
  selector: 
    app.kubernetes.io/instance: thanos-querier-example-thanos

links to

RHEA-2024:127118 Cluster Observability Operator 0.1 for OpenShift

rhobs/observability-operator#414: fix: use thanos default port in service and containerPort

1.	[QE] Add automation case for thano-querier to reach pod		Closed		Unassigned
2.	[QE] Add automation case for thano-querier to reach pod		Closed		Unassigned

Errata Tool added a comment - 2024/03/27 12:30 AM

Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

For information on the advisory (Cluster Observability Operator 0.1.2 for OpenShift), and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2024:1534

Errata Tool added a comment - 2024/03/27 12:30 AM Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Cluster Observability Operator 0.1.2 for OpenShift), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2024:1534

Hongyan Li added a comment - 2024/03/14 3:09 AM - edited

Test pass with upstream image quay.io/rhobs/observability-operator-bundle:0.1.0-rc

% oc -n e2e-tests get pod | grep thanos-querier-tq-ms
thanos-querier-tq-ms-combo-794b56d64f-v7jq4   1/1     Running   0          41s
% oc -n e2e-tests get svc | grep thanos-querier-tq-ms
thanos-querier-tq-ms-combo   ClusterIP   172.30.46.173   <none>        10902/TCP                    90s
% oc -n e2e-tests rsh thanos-querier-tq-ms-combo-794b56d64f-v7jq4

$ wget -O - 'http://172.30.46.173:10902/api/v1/query?query=prometheus_build_info'
Connecting to 172.30.46.173:10902 (172.30.46.173:10902)
writing to stdout
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"prometheus_build_info","branch":"HEAD","container":"prometheus","endpoint":"web","goarch":"amd64","goos":"linux","goversion":"go1.21.6","instance":"10.128.2.26:9090","job":"prometheus-self","namespace":"e2e-tests","pod":"prometheus-tq-ms-combo-0","prometheus":"e2e-tests/tq-ms-combo","revision":"43e14844a33b65e2a396e3944272af8b3a494071","service":"tq-ms-combo-prometheus","tags":"netgo,builtinassets,stringlabels","version":"2.49.1"},"value":[1710385468.868,"1"]},{"metric":{"__name__":"prometheus_build_info","branch":"HEAD","container":"prometheus","endpoint":"web","goarch":"amd64","goos":"linux","goversion":"go1.21.6","instance":"10.131.0.27:9090","job":"prometheus-self","namespace":"e2e-tests","pod":"prometheus-tq-ms-combo-1","prometheus":"e2e-tests/tq-ms-combo","revision":"43e14844a33b65e2a396e3944272af8b3a494071","service":"tq-ms-combo-prometheus","tags":"netgo,builtinassets,stringlabels","version":"2.49.1"},"value":[1710385468.868,"1"]}],"analysis":{}}}
$ wget -O - 'http://thanos-querier-tq-ms-combo.e2e-tests.svc:10902/api/v1/query?query=prometheus_build_info'
Connecting to thanos-querier-tq-ms-combo.e2e-tests.svc:10902 (172.30.46.173:10902)
writing to stdout
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"prometheus_build_info","branch":"HEAD","container":"prometheus","endpoint":"web","goarch":"amd64","goos":"linux","goversion":"go1.21.6","instance":"10.128.2.26:9090","job":"prometheus-self","namespace":"e2e-tests","pod":"prometheus-tq-ms-combo-0","prometheus":"e2e-tests/tq-ms-combo","revision":"43e14844a33b65e2a396e3944272af8b3a494071","service":"tq-ms-combo-prometheus","tags":"netgo,builtinassets,stringlabels","version":"2.49.1"},"value":[1710385819.867,"1"]},{"metric":{"__name__":"prometheus_build_info","branch":"HEAD","container":"prometheus","endpoint":"web","goarch":"amd64","goos":"linux","goversion":"go1.21.6","instance":"10.131.0.27:9090","job":"prometheus-self","namespace":"e2e-tests","pod":"prometheus-tq-ms-combo-1","prometheus":"e2e-tests/tq-ms-combo","revision":"43e14844a33b65e2a396e3944272af8b3a494071","service":"tq-ms-combo-prometheus","tags":"netgo,builtinassets,stringlabels","version":"2.49.1"},"value":[1710385819.867,"1"]}],"analysis":{}}}

Hongyan Li added a comment - 2024/03/14 3:09 AM - edited Test pass with upstream image quay.io/rhobs/observability-operator-bundle:0.1.0-rc % oc -n e2e-tests get pod | grep thanos-querier-tq-ms thanos-querier-tq-ms-combo-794b56d64f-v7jq4 1/1 Running 0 41s % oc -n e2e-tests get svc | grep thanos-querier-tq-ms thanos-querier-tq-ms-combo ClusterIP 172.30.46.173 <none> 10902/TCP 90s % oc -n e2e-tests rsh thanos-querier-tq-ms-combo-794b56d64f-v7jq4 $ wget -O - 'http: //172.30.46.173:10902/api/v1/query?query=prometheus_build_info' Connecting to 172.30.46.173:10902 (172.30.46.173:10902) writing to stdout { "status" : "success" , "data" :{ "resultType" : "vector" , "result" :[{ "metric" :{ "__name__" : "prometheus_build_info" , "branch" : "HEAD" , "container" : "prometheus" , "endpoint" : "web" , "goarch" : "amd64" , "goos" : "linux" , "goversion" : "go1.21.6" , "instance" : "10.128.2.26:9090" , "job" : "prometheus-self" , "namespace" : "e2e-tests" , "pod" : "prometheus-tq-ms-combo-0" , "prometheus" : "e2e-tests/tq-ms-combo" , "revision" : "43e14844a33b65e2a396e3944272af8b3a494071" , "service" : "tq-ms-combo-prometheus" , "tags" : "netgo,builtinassets,stringlabels" , "version" : "2.49.1" }, "value" :[1710385468.868, "1" ]},{ "metric" :{ "__name__" : "prometheus_build_info" , "branch" : "HEAD" , "container" : "prometheus" , "endpoint" : "web" , "goarch" : "amd64" , "goos" : "linux" , "goversion" : "go1.21.6" , "instance" : "10.131.0.27:9090" , "job" : "prometheus-self" , "namespace" : "e2e-tests" , "pod" : "prometheus-tq-ms-combo-1" , "prometheus" : "e2e-tests/tq-ms-combo" , "revision" : "43e14844a33b65e2a396e3944272af8b3a494071" , "service" : "tq-ms-combo-prometheus" , "tags" : "netgo,builtinassets,stringlabels" , "version" : "2.49.1" }, "value" :[1710385468.868, "1" ]}], "analysis" :{}}} $ wget -O - 'http: //thanos-querier-tq-ms-combo.e2e-tests.svc:10902/api/v1/query?query=prometheus_build_info' Connecting to thanos-querier-tq-ms-combo.e2e-tests.svc:10902 (172.30.46.173:10902) writing to stdout { "status" : "success" , "data" :{ "resultType" : "vector" , "result" :[{ "metric" :{ "__name__" : "prometheus_build_info" , "branch" : "HEAD" , "container" : "prometheus" , "endpoint" : "web" , "goarch" : "amd64" , "goos" : "linux" , "goversion" : "go1.21.6" , "instance" : "10.128.2.26:9090" , "job" : "prometheus-self" , "namespace" : "e2e-tests" , "pod" : "prometheus-tq-ms-combo-0" , "prometheus" : "e2e-tests/tq-ms-combo" , "revision" : "43e14844a33b65e2a396e3944272af8b3a494071" , "service" : "tq-ms-combo-prometheus" , "tags" : "netgo,builtinassets,stringlabels" , "version" : "2.49.1" }, "value" :[1710385819.867, "1" ]},{ "metric" :{ "__name__" : "prometheus_build_info" , "branch" : "HEAD" , "container" : "prometheus" , "endpoint" : "web" , "goarch" : "amd64" , "goos" : "linux" , "goversion" : "go1.21.6" , "instance" : "10.131.0.27:9090" , "job" : "prometheus-self" , "namespace" : "e2e-tests" , "pod" : "prometheus-tq-ms-combo-1" , "prometheus" : "e2e-tests/tq-ms-combo" , "revision" : "43e14844a33b65e2a396e3944272af8b3a494071" , "service" : "tq-ms-combo-prometheus" , "tags" : "netgo,builtinassets,stringlabels" , "version" : "2.49.1" }, "value" :[1710385819.867, "1" ]}], "analysis" :{}}}

Hongyan Li added a comment - 2024/02/04 4:31 AM - edited

Reproduced the issue:
Create monitoringstack and thanosquerier.

% oc -n e2e-tests get pod
NAME                                          READY   STATUS    RESTARTS   AGE   
thanos-querier-tq-ms-combo-57566f87bd-j6ssg   1/1     Running   0          6h26m
% oc -n e2e-tests get svc
NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
thanos-querier-tq-ms-combo   ClusterIP   172.30.44.137   <none>        9090/TCP                     6h26m
% oc -n e2e-tests rsh thanos-querier-tq-ms-combo-57566f87bd-j6ssg
sh-4.4$ curl -k 'http://172.30.44.137:9090/api/v1/query?' --data-urlencode 'query=prometheus_build_info'
curl: (7) Failed to connect to 172.30.44.137 port 9090: Connection refused
sh-4.4$ curl -k 'http://localhost:9090/api/v1/query?' --data-urlencode 'query=prometheus_build_info'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"prometheus_build_info","branch":"non-git","container":"prometheus","endpoint":"web","goarch":"amd64","goos":"linux","goversion":"go1.21.3","instance":"10.129.2.53:9090","job":"prometheus-self","namespace":"e2e-tests","pod":"prometheus-tq-ms-combo-1","prometheus":"e2e-tests/tq-ms-combo","revision":"non-git","service":"tq-ms-combo-prometheus","tags":"unknown","version":"2.48.1"},"value":[1707303226.118,"1"]},{"metric":{"__name__":"prometheus_build_info","branch":"non-git","container":"prometheus","endpoint":"web","goarch":"amd64","goos":"linux","goversion":"go1.21.3","instance":"10.131.0.21:9090","job":"prometheus-self","namespace":"e2e-tests","pod":"prometheus-tq-ms-combo-0","prometheus":"e2e-tests/tq-ms-combo","revision":"non-git","service":"tq-ms-combo-prometheus","tags":"unknown","version":"2.48.1"},"value":[1707303226.118,"1"]}],"analysis":{}}}

Hongyan Li added a comment - 2024/02/04 4:31 AM - edited Reproduced the issue: Create monitoringstack and thanosquerier. % oc -n e2e-tests get pod NAME READY STATUS RESTARTS AGE thanos-querier-tq-ms-combo-57566f87bd-j6ssg 1/1 Running 0 6h26m % oc -n e2e-tests get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE thanos-querier-tq-ms-combo ClusterIP 172.30.44.137 <none> 9090/TCP 6h26m % oc -n e2e-tests rsh thanos-querier-tq-ms-combo-57566f87bd-j6ssg sh-4.4$ curl -k 'http: //172.30.44.137:9090/api/v1/query?' --data-urlencode 'query=prometheus_build_info' curl: (7) Failed to connect to 172.30.44.137 port 9090: Connection refused sh-4.4$ curl -k 'http: //localhost:9090/api/v1/query?' --data-urlencode 'query=prometheus_build_info' { "status" : "success" , "data" :{ "resultType" : "vector" , "result" :[{ "metric" :{ "__name__" : "prometheus_build_info" , "branch" : "non-git" , "container" : "prometheus" , "endpoint" : "web" , "goarch" : "amd64" , "goos" : "linux" , "goversion" : "go1.21.3" , "instance" : "10.129.2.53:9090" , "job" : "prometheus-self" , "namespace" : "e2e-tests" , "pod" : "prometheus-tq-ms-combo-1" , "prometheus" : "e2e-tests/tq-ms-combo" , "revision" : "non-git" , "service" : "tq-ms-combo-prometheus" , "tags" : "unknown" , "version" : "2.48.1" }, "value" :[1707303226.118, "1" ]},{ "metric" :{ "__name__" : "prometheus_build_info" , "branch" : "non-git" , "container" : "prometheus" , "endpoint" : "web" , "goarch" : "amd64" , "goos" : "linux" , "goversion" : "go1.21.3" , "instance" : "10.131.0.21:9090" , "job" : "prometheus-self" , "namespace" : "e2e-tests" , "pod" : "prometheus-tq-ms-combo-0" , "prometheus" : "e2e-tests/tq-ms-combo" , "revision" : "non-git" , "service" : "tq-ms-combo-prometheus" , "tags" : "unknown" , "version" : "2.48.1" }, "value" :[1707303226.118, "1" ]}], "analysis" :{}}}

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Sub-Tasks

Activity

Collapse comment: Errata Tool added a comment - 2024/03/27 12:30 AM

Expand comment: Errata Tool added a comment - 2024/03/27 12:30 AM

Collapse comment: Hongyan Li added a comment - 2024/03/14 3:09 AM, Edited by Hongyan Li - 2024/03/14 3:11 AM

Expand comment: Hongyan Li added a comment - 2024/03/14 3:09 AM, Edited by Hongyan Li - 2024/03/14 3:11 AM

Collapse comment: Hongyan Li added a comment - 2024/02/04 4:31 AM, Edited by Hongyan Li - 2024/02/07 11:03 AM

Expand comment: Hongyan Li added a comment - 2024/02/04 4:31 AM, Edited by Hongyan Li - 2024/02/07 11:03 AM

People

Dates

PagerDuty