-
Bug
-
Resolution: Unresolved
-
Normal
-
Logging 6.4.0
-
Quality / Stability / Reliability
-
2
-
False
-
-
False
-
NEW
-
NEW
-
Release Note Not Required
-
-
-
Logging - Sprint 279
Description:
The logging-loki-loki-allow-bucket-egress NetworkPolicy generated by the LokiStack controller, it seems, does not allow egress to the configured NooBaa S3 in openshift-storage, causing ingestion and flush failures. The default policies created by the operator should be including configuration for ODF since this is a fairly common scenario.
Error:
failed to flush chunks: store put chunk:
RequestError: send request failed
caused by: dial tcp 172.30.174.166:443: i/o timeout
level=error ts=2025-11-03T03:25:36.67731524Z caller=flush.go:261 component=ingester loop=23 org_id=infrastructure msg="failed to flush" retries=1 err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://s3.openshift-storage.svc:443/kbhartimyloki/infrastructure/44d15f0c200e03f7/19a4734c4ca%3A19a47693239%3A17a03e0c\": dial tcp 172.30.174.166:443: i/o timeout, num_chunks: 1, labels: {k8s_container_name=\"kube-rbac-proxy-web\", k8s_namespace_name=\"openshift-monitoring\", k8s_node_name=\"ip-10-0-75-115.us-east-2.compute.internal\", k8s_pod_name=\"thanos-querier-6f757dd867-vl9cg\", kubernetes_container_name=\"kube-rbac-proxy-web\", kubernetes_host=\"ip-10-0-75-115.us-east-2.compute.internal\", kubernetes_namespace_name=\"openshift-monitoring\", kubernetes_pod_name=\"thanos-querier-6f757dd867-vl9cg\", log_type=\"infrastructure\", openshift_log_type=\"infrastructure\"}"
Bucket Egress network policy:
$ oc get networkpolicy logging-loki-loki-allow-bucket-egress -o yaml | yq -e .spec egress: - ports: - port: 443 protocol: TCP podSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - lokistack - key: app.kubernetes.io/component operator: In values: - ingester - querier - index-gateway - compactor - ruler policyTypes: - Egress
S3 service under openshift-storage namespace:
$ oc get svc -n openshift-storage | grep s3 s3 LoadBalancer 172.30.174.166 <hidden> 80:32533/TCP,443:31185/TCP,8444:32384/TCP,7004:31331/TCP 7h30m
Here the object storage secret odf-s3-secret contains AWS_ENDPOINT_URL as https://s3.openshift-storage.svc:443
LokiStack CR with networkPolicies enabled:
$ oc get lokistack logging-loki -o yaml | yq -e .spec managementState: Managed networkPolicies: disabled: false rules: enabled: false namespaceSelector: matchLabels: openshift.io/cluster-monitoring: "true" selector: matchLabels: openshift.io/cluster-monitoring: "true" size: 1x.demo storage: schemas: - effectiveDate: "2023-10-15" version: v13 secret: name: odf-s3-secret type: s3 tls: caName: lokistack-odf-ca storageClassName: myodf tenants: mode: openshift-logging
Steps to Reproduce:
a) Deploy ODF in MCG configuration
b) Create Loki Stack with network policies enabled.
Version: loki-operator.v6.4.0
How reproducible: Always
Expected Result: LokiStack should be able to forward to the NooBaa s3 backend since this is a fairly common scenario.
Actual Result: Timeout while forwarding to NooBaa s3 backend
Additional Info: Logs can be forwarded when networkPolicies is disabled.