Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2732

Prometheus Operator pod throws 'skipping servicemonitor' error on Jaeger integration

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, the Prometheus Operator skipped Elasticsearch Monitor for Jaeger setup caused missing metrics observability for Jaeger Elasticsearch instances. With this update, the Elasticsearch Operator reconciles proper ServiceMonitor Resources using the SafeTLSConfig resolves the issue of missing metrics for for Jaeger Elasticsearch instances.
      Show
      Before this update, the Prometheus Operator skipped Elasticsearch Monitor for Jaeger setup caused missing metrics observability for Jaeger Elasticsearch instances. With this update, the Elasticsearch Operator reconciles proper ServiceMonitor Resources using the SafeTLSConfig resolves the issue of missing metrics for for Jaeger Elasticsearch instances.
    • Logging (LogExp) - Sprint 220, Log Storage - Sprint 221, Log Storage - Sprint 222

    Description

      This bug is a followup on LOG-2696 for Logging 5.4.

      Error on Prometheus Operator pod

      [kbharti@cube ~]$ oc logs prometheus-operator-c7bdc5c48-dhnx2 | tail -n 5

level=warn ts=2022-06-15T13:13:51.759726324Z caller=operator.go:1832 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=jk-test/monitor-elasticsearch-cluster namespace=openshift-user-workload-monitoring prometheus=user-workload
level=info ts=2022-06-15T13:13:51.879904029Z caller=operator.go:643 component=thanosoperator key=openshift-user-workload-monitoring/user-workload msg="sync thanos-ruler"
level=info ts=2022-06-15T13:13:51.87999761Z caller=operator.go:1220 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
level=warn ts=2022-06-15T13:13:51.884746471Z caller=operator.go:1832 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=jk-test/monitor-elasticsearch-cluster namespace=openshift-user-workload-monitoring prometheus=user-workload
level=info ts=2022-06-15T13:13:51.934327802Z caller=operator.go:643 component=thanosoperator key=openshift-user-workload-monitoring/user-workload msg="sync thanos-ruler"

      Seems the labels match between service monitor and elasticsearch-metrics service but PrometheusOperatorRejectedResources Alert is being fired.

      Attached the servicemonitor and elasticsearch-metrics manifests.

      Steps to reproduce:

      1. Enable user-workload-monitoring
      2. Create a Jaeger CR as attached
      3. oc get servicemonitor -o yaml
      4. Observe if the matchLabels are correct
      5. Observe the prometheus-operator user-workload-monitoring pod and error of level=warn ts=<timestamp> caller=operator.go:1832 component=prometheusoperatormsg="skipping servicemonitor" error"it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=<namespace>/<name> namespace=openshift-user-workload-monitoring prometheus=user-workload

      CSV:

      [kbharti@cube ~]$ oc get csv
NAME                           DISPLAY                                          VERSION   REPLACES   PHASE
elasticsearch-operator.5.4.2   OpenShift Elasticsearch Operator                 5.4.2                Succeeded
jaeger-operator.v1.30.2        Red Hat OpenShift distributed tracing platform   1.30.2               Succeeded

       

       

      Attachments

        1. elasticsearch-metrics-svc.yaml
          1.0 kB
          Kabir Bharti
        2. jaeger.yaml
          0.8 kB
          Kabir Bharti
        3. servicemonitor.yaml
          2 kB
          Kabir Bharti

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. LOG-2845 Prometheus Operator pod throws 'skipping servicemonitor' error on Jaeger integration

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          links to

          GitHub openshift/elasticsearch-operator#903: LOG-2732: Fix ES servicemonitor for user-workload-monitoring

          mentioned on

          GitLab Merge request - Updated 3 upstream sources

          GitLab Merge request - Updated 5 upstream sources

          Activity

            People

              ptsiraki@redhat.com Periklis Tsirakidis
              rhn-support-kbharti Kabir Bharti
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: