-
Task
-
Resolution: Unresolved
-
Undefined
-
RHOAI_2.6.0, RHOAI_2.10.0
After upgrading to 2.6 I see following from the prometheus pod logs in openshift-monitoring namespace:
ts=2024-02-07T07:24:14.183Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:545: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:14.183Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:545: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:16.029Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:16.029Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"redhat-ods-applications\""{}ts=2024-02-07T07:24:27.010Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:544: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:27.011Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:544: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"redhat-ods-applications\""
Adding the following RoleBinding resolves the problem
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rhods-prometheus-cluster-monitoring-viewer-binding
namespace: redhat-ods-applications
subjects:
- kind: ServiceAccount
name: prometheus-k8s
namespace: openshift-monitoring
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-reader
There are more RoleBindings left in the redhat-ods-monitoring namespace that need cleanup.
oc get rolebinding -n redhat-ods-monitoring
NAME ROLE AGE
cluster-monitor-rhods-reader ClusterRole/cluster-reader 42h
openshift-pipelines-edit ClusterRole/edit 42h
pipelines-scc-rolebinding ClusterRole/pipelines-scc-clusterrole 42h
rhods-prometheus-cluster-monitoring-viewer-binding Role/rhods-prometheus-cluster-monitoring-viewer 42h
Please check.