Uploaded image for project: 'Red Hat OpenShift AI Engineering'
  1. Red Hat OpenShift AI Engineering
  2. RHOAIENG-2761

RoleBinding left in redhat-ods-monitoring namespace

XMLWordPrintable

    • Testable

      After upgrading to 2.6 I see following from the prometheus pod logs in openshift-monitoring namespace:

       

      ts=2024-02-07T07:24:14.183Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:545: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:14.183Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:545: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:16.029Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:16.029Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"redhat-ods-applications\""{}ts=2024-02-07T07:24:27.010Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:544: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"redhat-ods-applications\""ts=2024-02-07T07:24:27.011Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:544: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"redhat-ods-applications\""

       

      Adding the following RoleBinding resolves the problem

       

      kind: RoleBinding

      apiVersion: rbac.authorization.k8s.io/v1

      metadata:

        name: rhods-prometheus-cluster-monitoring-viewer-binding

        namespace: redhat-ods-applications

      subjects:

        - kind: ServiceAccount

           name: prometheus-k8s

           namespace: openshift-monitoring

      roleRef:

        apiGroup: rbac.authorization.k8s.io

        kind: ClusterRole

        name: cluster-reader

       

      There are more RoleBindings left in the redhat-ods-monitoring namespace that need cleanup.

      oc get rolebinding -n redhat-ods-monitoring       

      NAME                                                 ROLE                                              AGE

      cluster-monitor-rhods-reader                         ClusterRole/cluster-reader                        42h

      openshift-pipelines-edit                             ClusterRole/edit                                  42h

      pipelines-scc-rolebinding                            ClusterRole/pipelines-scc-clusterrole             42h

      rhods-prometheus-cluster-monitoring-viewer-binding   Role/rhods-prometheus-cluster-monitoring-viewer   42h

       

      Please check.

            rh-ee-wenzhou Wen Zhou
            rhn-support-afaulhab Anne Faulhaber
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: