-
Bug
-
Resolution: Unresolved
-
Normal
-
Logging 6.4.0
-
Quality / Stability / Reliability
-
3
-
False
-
-
False
-
NEW
-
NEW
-
Before the operator was exposing the incorrect port when networkpolicies where enabled. Now The operator now exposes the proxy port to allow communication between the ingester and objectstore through the proxy
-
Bug Fix
-
-
-
Logging - Sprint 280, Logging - Sprint 281
Description:
Egress to storage backend is not permitted when cluster-wide proxy is enabled.
Error:
level=error ts=2025-11-06T21:19:06.597204292Z caller=table_manager.go:143 index-store=tsdb-2023-10-15 msg="failed to upload table" table=index_20398 err="Post \"https://storage.googleapis.com/upload/storage/v1/b/kbhartimyloki/o?alt=json&name=index%2Findex_20398%2F1762458867-logging-loki-ingester-0-1762458867998644759.tsdb.gz&prettyPrint=false&projection=full&uploadType=multipart\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": proxyconnect tcp: dial tcp 10.0.0.2:3128: i/o timeout"
Cluster-wide proxy is enabled
$ oc get proxy cluster -o yaml | yq -e .status httpProxy: http://proxy-user2:<hidden>@10.0.0.2:3128 httpsProxy: http://proxy-user2:<hidden>@10.0.0.2:3128 noProxy: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.kbhartigcpx.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com
Proxy config under loki components:
$ oc get statefulset logging-loki-ingester -n openshift-logging -o json \ | jq -r '.spec.template.spec.containers[].env' [ { "name": "GOOGLE_APPLICATION_CREDENTIALS", "value": "/etc/storage/secrets/key.json" }, { "name": "HTTP_PROXY", "value": "http://proxy-user2:<hidden>@10.0.0.2:3128" }, { "name": "http_proxy", "value": "http://proxy-user2:<hidden>@10.0.0.2:3128" }, { "name": "HTTPS_PROXY", "value": "http://proxy-user2:<hidden>@10.0.0.2:3128" }, { "name": "https_proxy", "value": "http://proxy-user2:<hidden>@10.0.0.2:3128" }, { "name": "NO_PROXY", "value": ".cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.kbhartigcpx.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com" }, { "name": "no_proxy", "value": ".cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.kbhartigcpx.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com" } ]
Steps to Reproduce:
a) Deploy a cluster with cluster wide proxy enabled
b) Forward logs to storage backend, in this case GCS bucket
c) Enable network policies on lokistack CR
d) Observe Ingester pods logs
Version: loki-operator.v6.4.0
How reproducible: Always
Expected Result: LokiStack should be able to forward to the GCS bucket.
Actual Result: Timeout while forwarding to GCS bucket
Additional Info: Logs can be forwarded when networkPolicies is disabled. When NP is enabled and noProxy is patched with GCS API's, the flow works fine even with proxy enabled.
$ oc patch proxy cluster --type=merge -p '{
"spec": {
"noProxy": "test.no-proxy.com,storage.googleapis.com,oauth2.googleapis.com,googleapis.com"
}
}'