-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
RHODS_1.21.0_GA
Low
Description of problem:
Prometheus instances for model serving report many lines of error/warning about projects where no model is deployed.
While it could be expected, it may create noise in the logs
ts=2023-01-19T15:07:20.502Z caller=log.go:168 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:527: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:redhat-ods-monitoring:prometheus-custom\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"ods\"" ts=2023-01-19T15:07:20.502Z caller=log.go:168 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:527: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:redhat-ods-monitoring:prometheus-custom\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"ods\"" ts=2023-01-19T15:07:23.085Z caller=log.go:168 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:529: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:redhat-ods-monitoring:prometheus-custom\" cannot list resource \"pods\" in API group \"\" in the namespace \"ods\"" ts=2023-01-19T15:07:23.085Z caller=log.go:168 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:529: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:redhat-ods-monitoring:prometheus-custom\" cannot list resource \"pods\" in API group \"\" in the namespace \"ods\"" ts=2023-01-19T15:07:30.153Z caller=log.go:168 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:528: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:redhat-ods-monitoring:prometheus-custom\" cannot list resource \"services\" in API group \"\" in the namespace \"ods-2\"" ts=2023-01-19T15:07:30.153Z caller=log.go:168 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:528: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:redhat-ods-monitoring:prometheus-custom\" cannot list resource \"services\" in API group \"\" in the namespace \"ods-2\""
Prerequisites (if any, like setup, operators/versions):
Steps to Reproduce
- Create a DS Project from RHODS Dashboard (withoug deploying models or creating model servers)
- Go to OCP Console > Workloads > Pods > select NS redhat-ods-monitoring > open one of the three prometheus-rhods-model-monitoring- > Logs
- check the logs for any error/warning about the created DS Project
Actual results:
many error/warning lines about DS Projects where no model is deployed
Expected results:
there are not error/warning lines about DS Projects where no model is deployed
Reproducibility (Always/Intermittent/Only Once):
Always
Build Details:
RHODS v1.21.0-21