Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-8075

NetworkPolicy generated by the Loki Operator does not permit egress to the NooBaa S3 backend storage

XMLWordPrintable

    • Quality / Stability / Reliability
    • 3
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • NEW
    • Hide
      Before when a k8s service was configured for the object-store endpoint LO would just use the port of that service as the egressPort in the network policy. Now LO will fetch the Service and EndpointSlices and correctly expose the Pod Port rather than the Service Port
      Show
      Before when a k8s service was configured for the object-store endpoint LO would just use the port of that service as the egressPort in the network policy. Now LO will fetch the Service and EndpointSlices and correctly expose the Pod Port rather than the Service Port
    • Bug Fix
    • Logging - Sprint 279, Logging - Sprint 280

      Description:
      The logging-loki-loki-allow-bucket-egress NetworkPolicy generated by the LokiStack controller, it seems, does not allow egress to the configured NooBaa S3 in openshift-storage, causing ingestion and flush failures. The default policies created by the operator should be including configuration for ODF since this is a fairly common scenario.

      Error:
      failed to flush chunks: store put chunk:
      RequestError: send request failed
      caused by: dial tcp 172.30.174.166:443: i/o timeout

      level=error ts=2025-11-03T03:25:36.67731524Z caller=flush.go:261 component=ingester loop=23 org_id=infrastructure msg="failed to flush" retries=1 err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://s3.openshift-storage.svc:443/kbhartimyloki/infrastructure/44d15f0c200e03f7/19a4734c4ca%3A19a47693239%3A17a03e0c\": dial tcp 172.30.174.166:443: i/o timeout, num_chunks: 1, labels: {k8s_container_name=\"kube-rbac-proxy-web\", k8s_namespace_name=\"openshift-monitoring\", k8s_node_name=\"ip-10-0-75-115.us-east-2.compute.internal\", k8s_pod_name=\"thanos-querier-6f757dd867-vl9cg\", kubernetes_container_name=\"kube-rbac-proxy-web\", kubernetes_host=\"ip-10-0-75-115.us-east-2.compute.internal\", kubernetes_namespace_name=\"openshift-monitoring\", kubernetes_pod_name=\"thanos-querier-6f757dd867-vl9cg\", log_type=\"infrastructure\", openshift_log_type=\"infrastructure\"}"

      Bucket Egress network policy:

      $ oc get networkpolicy logging-loki-loki-allow-bucket-egress -o yaml | yq -e .spec
egress:
  - ports:
      - port: 443
        protocol: TCP
podSelector:
  matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values:
        - lokistack
    - key: app.kubernetes.io/component
      operator: In
      values:
        - ingester
        - querier
        - index-gateway
        - compactor
        - ruler
policyTypes:
  - Egress

      S3 service under openshift-storage namespace:

      $ oc get svc -n openshift-storage | grep s3
s3                                                LoadBalancer   172.30.174.166   <hidden>   80:32533/TCP,443:31185/TCP,8444:32384/TCP,7004:31331/TCP   7h30m

      Here the object storage secret odf-s3-secret contains AWS_ENDPOINT_URL as https://s3.openshift-storage.svc:443

      LokiStack CR with networkPolicies enabled:

       

      $ oc get lokistack logging-loki -o yaml | yq -e .spec
managementState: Managed
networkPolicies:
  disabled: false
rules:
  enabled: false
  namespaceSelector:
    matchLabels:
      openshift.io/cluster-monitoring: "true"
  selector:
    matchLabels:
      openshift.io/cluster-monitoring: "true"
size: 1x.demo
storage:
  schemas:
    - effectiveDate: "2023-10-15"
      version: v13
  secret:
    name: odf-s3-secret
    type: s3
  tls:
    caName: lokistack-odf-ca
storageClassName: myodf
tenants:
  mode: openshift-logging
 

      Steps to Reproduce:
      a) Deploy ODF in MCG configuration
      b) Create Loki Stack with network policies enabled.

      Version: loki-operator.v6.4.0

      How reproducible: Always

      Expected Result: LokiStack should be able to forward to the NooBaa s3 backend since this is a fairly common scenario. 

      Actual Result: Timeout while forwarding to NooBaa s3 backend

      Additional Info: Logs can be forwarded when networkPolicies is disabled.

              jmarcal@redhat.com Joao Marcal
              rhn-support-kbharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: