-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.16, 4.18
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
The OpenShift Update Service has not set the needed permissions for the `system:serviceaccount:openshift-monitoring:prometheus-k8s` being able to scrape the resources for getting the metrics in the namespace `openshift-update-service` throwing the Prometheus pods the error: $ oc logs prometheus-k8s-0 -n openshift-monitoring|grep "openshift-update-service" |tail -2 ts=2025-02-24T11:25:18.124Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\"" ts=2025-02-24T11:25:18.124Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\""
Version-Release number of selected component (if applicable):
OpenShift Update Service Operator 5.0.3 latest version OpenShift 4.16.36
How reproducible:
Always
Steps to Reproduce:
1. Install OpenShift 4.16.z latest version (4.16.35) 2. Install the Openshift Update Service Operator 5.y latest version 3. Add the label "openshift.io/cluster-monitoring=true" to the namespace "openshift-update-service" if it was installed using the OpenShift Console $ oc label ns/openshift-update-service openshift.io/cluster-monitoring=true
Actual results:
Verify that the OpenShift Prometheus Core pods are not able to scrape the metrics exposed by the resources in the "openshift-update-service" and it's observed in the OpenShift Core Prometheus the error: $ oc logs prometheus-k8s-0 -n openshift-monitoring|grep "openshift-update-service" |tail -2 ts=2025-02-24T11:25:18.124Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\"" ts=2025-02-24T11:25:18.124Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\""
Expected results:
The OpenShift Update Service Operator manages the creation of the permissions needed for the metrics exposed for the resources created by this Operator being scrapped by the OpenShift Prometheus, the same that it's done by other Operators as the Red Hat Logging Operator
Workaround
Create a role using the yaml below: $ cat role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-k8s namespace: openshift-update-service rules: - apiGroups: - "" resources: - services - endpoints - pods verbs: - get - list - watch Create the RoleBinding for allowing to the serviceAccount prometheus-k8s scrape the metrics from the resources in the namespace "openshift-update-service" using the yaml: $ cat rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s namespace: openshift-update-service roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: prometheus-k8s subjects: - kind: ServiceAccount name: prometheus-k8s namespace: openshift-monitoring
- relates to
-
OCPBUGS-60905 Operator metrics service does not match operator pods
-
- New
-
- links to