Uploaded image for project: 'Cluster Observability Operator'
  1. Cluster Observability Operator
  2. COO-1327

Prometheus operator fails to process PodMonitors with label replacements

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 1.3.0
    • monitoring-stack
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Started after upgrading COO to 1.3.0

      ts=2025-11-13T20:59:24.708174781Z level=error caller=/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.34.1/tools/events/event_broadcaster.go:270 msg="Server rejected event (will not retry!)" err="events.events.k8s.io is forbidden: User \"system:serviceaccount:openshift-operators:obo-prometheus-operator\" cannot create resource \"events\" in API group \"events.k8s.io\" in the namespace \"rhacs-observability\"" event="&Event\{ObjectMeta:{rhacs-fleetshard-metrics.1877ac9be55cb741 rhacs-observability 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},EventTime:2025-11-13 20:59:24.706802847 +0000 UTC m=+4113.238025722,Series:nil,ReportingController:prometheus-controller,ReportingInstance:prometheus-controller-obo-prometheus-operator-846dcc4684-v55gg,Action:SelectingConfigurationResources,Reason:InvalidConfiguration,Regarding:\{PodMonitor rhacs-observability rhacs-fleetshard-metrics 1eec3722-e35b-4716-97c7-7a5867a3351d monitoring.rhobs/v1 25053409 },Related:&v11.ObjectReference\{Kind:Prometheus,Namespace:rhacs-observability,Name:obs-prometheus,UID:d0b31036-c450-44e2-b727-a7f1a0dbf382,APIVersion:monitoring.rhobs/v1,ResourceVersion:162923755,FieldPath:,},Note:\"rhacs-observability/rhacs-fleetshard-metrics\" was rejected due to invalid configuration: endpoint[0]: relabelConfigs: [2]: \"rhacs_${1}\" is invalid 'replacement' for labelmap action,Type:Warning,DeprecatedSource:\{ },DeprecatedFirstTimestamp:0001-01-01 00:00:00 +0000 UTC,DeprecatedLastTimestamp:0001-01-01 00:00:00 +0000 UTC,DeprecatedCount:0,}"
      

      Prometheus-operator version: 0.86.1.
      Caused by the upstream bug: https://github.com/prometheus-operator/prometheus-operator/issues/8041

        1. podmonitor.sh
          3 kB
          Hongyan Li
        2. workaround.sh
          0.6 kB
          Hongyan Li

              Unassigned Unassigned
              ykovalev@redhat.com Yury Kovalev
              None
              None
              None
              Hongyan Li Hongyan Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: