Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-8084

Egress to storage backend is timed out when cluster-wide proxy is enabled

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • NEW
    • Bug Fix
    • Logging - Sprint 280

      Description:
      Egress to storage backend is not permitted when cluster-wide proxy is enabled.

      Error:

      level=error ts=2025-11-06T21:19:06.597204292Z caller=table_manager.go:143 index-store=tsdb-2023-10-15 msg="failed to upload table" table=index_20398 err="Post \"https://storage.googleapis.com/upload/storage/v1/b/kbhartimyloki/o?alt=json&name=index%2Findex_20398%2F1762458867-logging-loki-ingester-0-1762458867998644759.tsdb.gz&prettyPrint=false&projection=full&uploadType=multipart\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": proxyconnect tcp: dial tcp 10.0.0.2:3128: i/o timeout"

       

      Cluster-wide proxy is enabled

      $ oc get proxy cluster -o yaml | yq -e .status
httpProxy: http://proxy-user2:<hidden>@10.0.0.2:3128
httpsProxy: http://proxy-user2:<hidden>@10.0.0.2:3128
noProxy: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.kbhartigcpx.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com

       

      Proxy config under loki components:

      $ oc get statefulset logging-loki-ingester -n openshift-logging -o json \
  | jq -r '.spec.template.spec.containers[].env' 
[
  {
    "name": "GOOGLE_APPLICATION_CREDENTIALS",
    "value": "/etc/storage/secrets/key.json"
  },
  {
    "name": "HTTP_PROXY",
    "value": "http://proxy-user2:<hidden>@10.0.0.2:3128"
  },
  {
    "name": "http_proxy",
    "value": "http://proxy-user2:<hidden>@10.0.0.2:3128"
  },
  {
    "name": "HTTPS_PROXY",
    "value": "http://proxy-user2:<hidden>@10.0.0.2:3128"
  },
  {
    "name": "https_proxy",
    "value": "http://proxy-user2:<hidden>@10.0.0.2:3128"
  },
  {
    "name": "NO_PROXY",
    "value": ".cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.kbhartigcpx.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com"
  },
  {
    "name": "no_proxy",
    "value": ".cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.kbhartigcpx.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com"
  }
]

       

      Steps to Reproduce:
      a) Deploy a cluster with cluster wide proxy enabled
      b) Forward logs to storage backend, in this case GCS bucket
      c) Enable network policies on lokistack CR
      d) Observe Ingester pods logs

      Version: loki-operator.v6.4.0

      How reproducible: Always

      Expected Result: LokiStack should be able to forward to the GCS bucket.

      Actual Result: Timeout while forwarding to GCS bucket

      Additional Info: Logs can be forwarded when networkPolicies is disabled. When NP is enabled and noProxy is patched with GCS API's, the flow works fine even with proxy enabled.

       

      $ oc patch proxy cluster --type=merge -p '{
  "spec": {
    "noProxy": "test.no-proxy.com,storage.googleapis.com,oauth2.googleapis.com,googleapis.com"
  }
}'

       

              Unassigned Unassigned
              rhn-support-kbharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: