-
Bug
-
Resolution: Unresolved
-
Major
-
3.19.0.GA, 3.18.1.GA
-
None
-
False
-
-
False
-
-
-
Moderate
Description of problem:
The Red Hat Dev Spaces operator can be installed in any namespace that the OpenShift admin chooses as documented in [0] when it's said:
> The Red Hat OpenShift Dev Spaces Operator depends on the Dev Workspace Operator. If you install the Red Hat OpenShift Dev Spaces Operator manually to a non-default namespace, ensure that the Dev Workspace Operator is also installed in the same namespace. This is required as the Operator Lifecycle Manager will attempt to install the Dev Workspace Operator as a dependency within the Red Hat OpenShift Dev Spaces Operator namespace, potentially resulting in two conflicting installations of the Dev Workspace Operator if the latter is installed in a different namespace.
Then, the only requirement for when installing the OpenShift Dev Spaces Operator in not the default namespace: "openshift-devspaces" is to be sure that the Dev Workspace Operator is also installed in the same namespace.
After saying this, if the Red Hat Dev Spaces Operator is installed in a custom namespace, it's observed in the Prometheus pods the next error:
ts=2025-04-04T15:38:28.201Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:554: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"openshift-devspaces\""
Reviewing the "serviceMonitor: openshift-devspaces-metrics-export" is observed that it has coded to watch the "openshift-operators" and "openshift-devspaces", then failing with this second as the operator is not installed in the default namespace:
$ oc get serviceMonitor NAME AGE openshift-devspaces-metrics-exporter 92s $ oc get serviceMonitor -o yaml apiVersion: v1 items: - apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: creationTimestamp: "2025-04-07T09:46:56Z" generation: 2 labels: olm.managed: "true" name: openshift-devspaces-metrics-exporter namespace: testdevspaces ownerReferences: - apiVersion: operators.coreos.com/v1alpha1 blockOwnerDeletion: false controller: false kind: ClusterServiceVersion name: devspacesoperator.v3.19.0 uid: eff8167f-39cb-4f4e-af3e-c64b7f05cf49 resourceVersion: "210802" uid: 14aa1de5-ae50-4137-92af-1001973c5c4b spec: endpoints: - interval: 30s port: metrics scheme: http namespaceSelector: matchNames: - openshift-devspaces - openshift-operators selector: matchLabels: app.kubernetes.io/component: devspaces kind: List metadata: resourceVersion: ""
Prerequisites (if any, like setup, operators/versions):
$ oc get csv -n testdevspaces |grep ^dev devspacesoperator.v3.19.0 Red Hat OpenShift Dev Spaces 3.19.0 devspacesoperator.v3.18.1-0.1739475915.p Succeeded devworkspace-operator.v0.32.1 DevWorkspace Operator 0.32.1 devworkspace-operator.v0.31.2 Succeeded
Steps to Reproduce
1. Install the Red Hat Dev Spaces operator in a custom namespace, as it could be "testdevspaces"
2. Label the namespace to be monitored
oc label ns/testdevspaces openshift.io/cluster-monitoring=true
Actual results:
Verify that the serviceMonitor "openshift-devspaces-metrics-exporter" is matching to scrape the "openshift-operators" and the "openshift-devspaces" and as this second doesn't exist, the Prometheus pods returns the error:
$ oc logs prometheus-k8s-0 -n openshift-monitoring|grep devspaces|tail -2 ts=2025-04-07T10:02:36.189Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-devspaces\"" ts=2025-04-07T10:02:36.189Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:556: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-devspaces\""
Expected results:
Expected to have fixed:
- https://issues.redhat.com/browse/CRW-8315 where the needed serviceAccounts are created by default
- As it's supported to deploy in a custom namespace, the serviceMonitor monitors this custom namespace and not trying to monitor the "openshift-devspaces" namespace
Reproducibility (Always/Intermittent/Only Once):
Always
Acceptance criteria:
After deploying the Red Hat OpenShift Dev Spaces Operator is a custom namespace, the metrics are scrapped from this custom namespace and the Prometheus not returning an error as not able to scrape from the namespace "openshift-devspaces"
Definition of Done:
