-
Bug
-
Resolution: Done-Errata
-
Undefined
-
None
-
4.16.z
Description of problem:
The cluster-wide proxy is getting injected for remote-write config automatically but not the noProxy URLs in Prometheus k8s CR which is available in openshift-monitoring project which is expected. However, if the remote-write endpoint is in noProxy region, then metrics are not transferred.
Version-Release number of selected component (if applicable):
RHOCP 4.16.4
How reproducible:
100%
Steps to Reproduce:
1. Configure proxy custom resource in RHOCP 4.16.4 cluster
2. Create cluster-monitoring-config configmap in openshift-monitoring project
3. Inject remote-write config (without specifically configuring proxy for remote-write)
4. After saving the modification in cluster-monitoring-config configmap, check the remoteWrite config in Prometheus k8s CR. Now it contains the proxyUrl but NOT the noProxy URL(referenced from cluster proxy). Example snippet:
==============
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
[...]
name: k8s
namespace: openshift-monitoring
spec:
[...]
remoteWrite:
- proxyUrl: http://proxy.abc.com:8080 <<<<<====== Injected Automatically but there is no noProxy URL.
url: http://test-remotewrite.test.svc.cluster.local:9090
Actual results:
The proxy URL from proxy CR is getting injected in Prometheus k8s CR automatically when configuring remoteWrite but it doesn't have noProxy inherited from cluster proxy resource.
Expected results:
The noProxy URL should get injected in Prometheus k8s CR as well.
Additional info:
- blocks
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- is related to
-
-
- Verified
-
- is triggering
-
MON-4195 Corrective Measure for OCPBUGS-38289: [CEE.neXT]noProxy URL not available in Prometheus k8s CR after configuring remote-write
-
- Closed
-
-
MON-3991 Set noProxy in remote-write config via Prometheus CR
-
- Closed
-
- links to
-
RHEA-2024:6122
OpenShift Container Platform 4.18.z bug fix update
[OCPBUGS-38289] [CEE.neXT]noProxy URL not available in Prometheus k8s CR after configuring remote-write
rhn-support-tsaito you should follow the cloned tickets:
- 4.16.z https://issues.redhat.com/browse/OCPBUGS-39170
- 4.15.z https://issues.redhat.com/browse/OCPBUGS-39172
- 4.14.z https://issues.redhat.com/browse/OCPBUGS-39176
All fixes are merged but we need to wait for the next z stream releases to be published.
OpenShift Jira Bot
added a comment - Hi rh-ee-amrini,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
OpenShift Jira Bot
added a comment - Hi rh-ee-amrini ,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified. rhn-support-dgautam yes putting CMO into unmanaged mode and modifying the Prometheus object by hand can be a workaround.
Maximilian Grabowski (Inactive)
added a comment - Hey, I am just commenting because i originally opened the Case with the support.
We have a Thanos running inside the same cluster for long-term storage of prometheus data and want to reach it over the service network using the .svc.cluster.local domain, which is not possible anymore. The proxy variable gets injected, but not the no_proxy variable, sending all the prometheus data to our internet proxy which is unable to reach the internal Thanos URL. Manually specifying noProxy or no_proxy inside the cluster-monitoring-config seems to not be recognized by the cluster-monitoring operator.
Maximilian Grabowski (Inactive)
added a comment - Hey, I am just commenting because i originally opened the Case with the support.
We have a Thanos running inside the same cluster for long-term storage of prometheus data and want to reach it over the service network using the .svc.cluster.local domain, which is not possible anymore. The proxy variable gets injected, but not the no_proxy variable, sending all the prometheus data to our internet proxy which is unable to reach the internal Thanos URL. Manually specifying noProxy or no_proxy inside the cluster-monitoring-config seems to not be recognized by the cluster-monitoring operator. Hello Dhruv,
Thanks for reporting this.
Could you tell us more, why shouldn't the proxying be set up by default if the cluster wide proxy is set? Does this break something?
Even though it isn't explicitly documented (maybe I'm missing sth), the default proxying was supposed to be set up for remote-write as well https://github.com/openshift/cluster-monitoring-operator/pull/617/files as it's the case for telemetry client https://github.com/openshift/cluster-monitoring-operator/blob/c9a1d8d649083ce3ac717de78e9391146b5548f1/pkg/manifests/manifests.go#L2967-L2981
But because of a bug, that setup was no-op until https://github.com/openshift/cluster-monitoring-operator/pull/2269 (fix released in 4.16.0).
At least, we should document this better.
cc jfajersk@redhat.comspasquie@redhat.com
—
That being said, I think users cannot override that using "RemoteWriteSpec.proxyUrl" because of the config precedence here https://github.com/openshift/cluster-monitoring-operator/blob/c9a1d8d649083ce3ac717de78e9391146b5548f1/pkg/manifests/manifests.go#L1420-L1433 (default proxy setup always has the last word) and I think that should be fixed.
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Important: OpenShift Container Platform 4.18.1 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:6122