Uploaded image for project: 'OpenShift GitOps'
  1. OpenShift GitOps
  2. GITOPS-7992

openshift-gitops-operator-metrics-monitor ServiceMonitor is attempting to use a bearerTokenFile configuration in its endpoints definition

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 1.17.0, 1.18.0
    • Operator
    • False
    • Hide

      None

      Show
      None
    • False

      Description of Problem

      • The openshift-gitops-operator-metrics-monitor ServiceMonitor, automatically created by the OpenShift GitOps Operator (v1.18.0), is being rejected by the Prometheus Operator in the openshift-user-workload-monitoring namespace. This rejection prevents the collection of metrics for the GitOps Operator via the User Workload Monitoring stack and triggers PrometheusOperatorRejectedResources alerts. The Prometheus Operator logs indicate the rejection is due to the ServiceMonitor attempting to access file system via bearer token file, which is prohibited by the Prometheus specification and/or the UWM configuration.

      Additional Info

      • Prometheus Operator Logs (from openshift-user-workload-monitoring):

      ---

      {{ts=2025-10-01T23:41:03.20280244Z level=warn caller=/go/src/github.com/coreos/prometheus-operator/pkg/prometheus/resource_selector.go:133 msg="skipping servicemonitor" component=prometheus-controller error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=openshift-gitops-operator/openshift-gitops-operator-metrics-monitor namespace=openshift-gitops-operator prometheus=user-workload
      ts=2025-10-01T23:41:03.202944063Z level=info caller=/go/src/github.com/coreos/prometheus-operator/vendor/k8s.io/client-go/tools/record/event.go:389 msg="Event occurred" object.name=openshift-gitops-operator-metrics-monitor object.namespace=openshift-gitops-operator fieldPath="" kind=ServiceMonitor apiVersion=monitoring.coreos.com/v1 type=Warning reason=InvalidConfiguration message="ServiceMonitor openshift-gitops-operator-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits"
      }}

      ---

      • Alert Firing: An alert named {{PrometheusOperatorRejectedResources with severity Warning is continuously firing.}}

      Problem Reproduction

      • The problem consistently reproduces after the deployment of the OpenShift GitOps Operator. The ServiceMonitor is automatically created by the operator, leading to its immediate rejection by the Prometheus Operator.

      Reproducibility

      • Always

      Prerequisites/Environment

      • OpenShift Version: 4.18
      • OpenShift GitOps Operator Version: v1.18.0 (as per CSV in ownerReferences)
      • Prometheus Operator Version: 0.78.2 (as seen in Prometheus Operator logs)
      • Monitoring Configuration: User Workload Monitoring is enabled and managing monitoring resources in user-defined namespaces.

      Steps to Reproduce

      • Ensure an OpenShift cluster with User Workload Monitoring enabled.
      • Install the OpenShift GitOps Operator (e.g., via OperatorHub) to version v1.18.0.
      • Verify the openshift-gitops-operator namespace exists and contains the GitOps operator's pods.
      • Observe the creation of the openshift-gitops-operator-metrics-monitor ServiceMonitor in the openshift-gitops-operator namespace.
      • Check the logs of the Prometheus Operator pod in openshift-user-workload-monitoring (oc logs -n openshift-user-workload-monitoring -l app.kubernetes.io/name=prometheus-operator).

      Expected Results

      • The openshift-gitops-operator-metrics-monitor ServiceMonitor should be successfully accepted by the Prometheus Operator, allowing metrics for the GitOps Operator to be scraped and displayed within the OpenShift monitoring stack. No PrometheusOperatorRejectedResources alert should be firing for this ServiceMonitor.

      Actual Results

      • The openshift-gitops-operator-metrics-monitor ServiceMonitor is rejected by the Prometheus Operator due to its use of bearerTokenFile, leading to metrics not being scraped and the PrometheusOperatorRejectedResources alert continuously firing.

      Problem Analysis

      • <Completed by engineering team as part of the triage/refinement process>

      Root Cause

      • <What is the root cause of the problem? Or, why is it not a bug?>

      Workaround (If Possible)

      • <Are there any workarounds we can provide to the customers?>

      Fix Approaches

      • <If we decide to fix this bug, how will we do it?>

      Acceptance Criteria

      • ...

      Definition of Done

      • Code Complete:
        • All code has been written, reviewed, and approved.
      • Tested:
        • Unit tests have been written and passed.
        • Ensure code coverage is not reduced with the changes.
        • Integration tests have been automated.
        • System tests have been conducted, and all critical bugs have been fixed.
        • Tested and merged on OpenShift either upstream or downstream on a local build.
      • Documentation:
        • User documentation or release notes have been written (if applicable).
      • Build:
        • Code has been successfully built and integrated into the main repository / project.
        • Midstream changes (if applicable) are done, reviewed, approved and merged.
      • Review:
        • Code has been peer-reviewed and meets coding standards.
        • All acceptance criteria defined in the user story have been met.
        • Tested by reviewer on OpenShift.
      • Deployment:
        • The feature has been deployed on OpenShift cluster for testing.

              rh-ee-sghadi Siddhesh Ghadi
              llevy Leon Levy
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: