Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-51196

Not set permissions to system:serviceaccount:openshift-monitoring:prometheus-k8s for scrapping the `openshift-update-service` namespace resources

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      The OpenShift Update Service has not set the needed permissions for the `system:serviceaccount:openshift-monitoring:prometheus-k8s` being able to scrape the resources for getting the metrics in the namespace `openshift-update-service` throwing the Prometheus pods the error:
      
      $ oc logs prometheus-k8s-0 -n openshift-monitoring|grep "openshift-update-service" |tail -2
      ts=2025-02-24T11:25:18.124Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\""
      ts=2025-02-24T11:25:18.124Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\""

      Version-Release number of selected component (if applicable):

      OpenShift Update Service Operator 5.0.3 latest version
      OpenShift 4.16.36

      How reproducible:

      Always

      Steps to Reproduce:

      1. Install OpenShift 4.16.z latest version (4.16.35)
      2. Install the Openshift Update Service Operator 5.y latest version
      3. Add the label "openshift.io/cluster-monitoring=true" to the namespace "openshift-update-service" if it was installed using the OpenShift Console
      
          $ oc label ns/openshift-update-service openshift.io/cluster-monitoring=true     

      Actual results:

      Verify that the OpenShift Prometheus Core pods are not able to scrape the metrics exposed by the resources in the "openshift-update-service" and it's observed in the OpenShift Core Prometheus the error:
      
      $ oc logs prometheus-k8s-0 -n openshift-monitoring|grep "openshift-update-service" |tail -2
      ts=2025-02-24T11:25:18.124Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\"" ts=2025-02-24T11:25:18.124Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-update-service\""    

      Expected results:

      The OpenShift Update Service Operator manages the creation of the permissions needed for the metrics exposed for the resources created by this Operator being scrapped by the OpenShift Prometheus, the same that it's done by other Operators as the Red Hat Logging Operator

      Workaround

      Create a role using the yaml below:
      
      $ cat role.yaml 
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: prometheus-k8s
        namespace: openshift-update-service
      rules:
      - apiGroups:
        - ""
        resources:
        - services
        - endpoints
        - pods
        verbs:
        - get
        - list
        - watch
      
      Create the RoleBinding for allowing to the serviceAccount prometheus-k8s scrape the metrics from the resources in the namespace "openshift-update-service" using the yaml:
      
      $ cat rolebinding.yaml 
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: prometheus-k8s
        namespace: openshift-update-service 
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: prometheus-k8s
      subjects:
      - kind: ServiceAccount
        name: prometheus-k8s
        namespace: openshift-monitoring

              Unassigned Unassigned
              rhn-support-ocasalsa Oscar Casal Sanchez
              None
              None
              Jia Liu Jia Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: