Uploaded image for project: 'Cluster Observability Operator'
  1. Cluster Observability Operator
  2. COO-1114

thanosquerier failed to find monitoringstack which is added label after thanos-querier created

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 1.2.2
    • monitoring-stack
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None

      observability-operator has reconcile issue for thanos-querier, thanosquerier failed to find monitoringstack which is added label after thanos-querier created.

      Detail steps:

      1. Install COO 1.2.1 

      2. create monitoringstack, example app and thanos-querier

      change the following line to

       

      testSucceeded := !t.Failed()

       

      execute command

       

      go test -v -failfast ./test/e2e/... --retain=true -run TestMonitoringStackController/Verify_multi-namespace_support

       

      hongyli@hongyli-mac upgrade % cat <<EOF|oc create -f -
apiVersion: monitoring.rhobs/v1alpha1
kind: ThanosQuerier
metadata:
  name: example-thanos
  namespace: e2e-tests
spec:
  selector:
    matchLabels:
      mso: example
---
EOF


thanosquerier.monitoring.rhobs/example-thanos created
hongyli@hongyli-mac upgrade % oc -n e2e-tests label monitoringstack multi-ns  mso=example
oc -n e2e-tests get pod


monitoringstack.monitoring.rhobs/multi-ns labeled
NAME                                           READY   STATUS    RESTARTS   AGE
alertmanager-multi-ns-0                        2/2     Running   0          4m57s
alertmanager-multi-ns-1                        2/2     Running   0          4m57s
prometheus-multi-ns-0                          3/3     Running   0          4m57s
prometheus-multi-ns-1                          3/3     Running   0          4m57s
thanos-querier-example-thanos-575554d6-c2ppj   1/1     Running   0          9s
      hongyli@hongyli-mac upgrade % oc -n e2e-tests exec deploy/thanos-querier-example-thanos -- curl -k 'http://thanos-querier-example-thanos.e2e-tests.svc:10902/api/v1/query?' --data-urlencode 'query=version' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    90  100    77  100    13  15400   2600 --:--:-- --:--:-- --:--:-- 18000
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [],
    "analysis": {}
  }
}

      wait for about an hour and check data, still no data and check logs

       

       

      % oc logs observability-operator-8f498584d-r6ps2
.....
2025-07-23T03:16:31Z	INFO	observability-operator	Failed to update status	{"stack": {"name":"multi-ns","namespace":"e2e-tests"}, "err": "Operation cannot be fulfilled on monitoringstacks.monitoring.rhobs \"multi-ns\": the object has been modified; please apply your changes to the latest version and try again"}
      % oc logs observability-operator-8f498584d-r6ps2 | grep thanos
.....
2025-07-23T02:05:52Z	INFO	Starting Controller	{"controller": "thanosquerier", "controllerGroup": "monitoring.rhobs", "controllerKind": "ThanosQuerier"}
2025-07-23T02:05:52Z	INFO	Starting workers	{"controller": "thanosquerier", "controllerGroup": "monitoring.rhobs", "controllerKind": "ThanosQuerier", "worker count": 1}
2025-07-23T02:35:41Z	INFO	thanos-querier	watched MonitoringStack changed, checking for matching querier	{"Monitoring Stack": "e2e-tests/multi-ns"}
2025-07-23T02:40:30Z	INFO	thanos-querier	Reconciling Thanos Querier	{"querier": {"name":"example-thanos","namespace":"e2e-tests"}}
2025-07-23T02:40:30Z	INFO	thanos-querier	Found MonitoringStacks list	{"selector": {"matchLabels":{"mso":"example"}}, "length": 0}
2025-07-23T02:40:30Z	INFO	thanos-querier	Reconciling Thanos Querier	{"querier": {"name":"example-thanos","namespace":"e2e-tests"}}
2025-07-23T02:40:30Z	INFO	thanos-querier	Found MonitoringStacks list	{"selector": {"matchLabels":{"mso":"example"}}, "length": 0}

      delete thanos-querier and create again

      % oc logs observability-operator-8f498584d-r6ps2 | grep thanos
.....
2025-07-23T04:13:31Z	INFO	thanos-querier	Found MonitoringStacks list	{"selector": {"matchLabels":{"mso":"example"}}, "length": 1}
2025-07-23T04:13:31Z	INFO	thanos-querier	Reconciling Thanos Querier	{"querier": {"name":"example-thanos","namespace":"e2e-tests"}}
2025-07-23T04:13:31Z	INFO	thanos-querier	Found MonitoringStacks list	{"selector": {"matchLabels":{"mso":"example"}}, "length": 1}    

% oc -n e2e-tests exec deploy/thanos-querier-example-thanos -- curl -k 'http://thanos-querier-example-thanos.e2e-tests.svc:10902/api/v1/query?' --data-urlencode 'query=version' | jq | head -n 25
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1064  100  1051  100    13   146k   1857 --:--:-- --:--:-- --:--:--  148k
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "version",
          "container": "prometheus-example-app",
          "endpoint": "metrics",
          "instance": "10.130.2.26:8080",
          "job": "prometheus-example-app",
          "namespace": "test-ns-3",
          "pod": "prometheus-example-app",
          "prometheus": "e2e-tests/multi-ns",
          "service": "prometheus-example-app",
          "version": "v0.4.1"
        },
        "value": [
          1753245597.537,
          "1"
        ]
      },
      {
        "metric": {

       

       

              jfajersk@redhat.com Jan Fajerski
              hongyli@redhat.com Hongyan Li
              None
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: