-
Bug
-
Resolution: Unresolved
-
Blocker
-
Logging 6.5.0
-
Quality / Stability / Reliability
-
3
-
False
-
-
False
-
NEW
-
NEW
-
Bug Fix
-
-
-
Logging - Sprint 285
Description of problem:
When Loki Operator is provisioned in CCO mode on AWS STS cluster, compactor and ingestor components throw Invalid Configuration: Missing Region in ingester storage/STS endpoint resolution
Compactor logs:
init compactor: failed to init delete store: failed to get s3 object: operation error S3: GetObject, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region error initialising module: compactor github.com/grafana/dskit/modules.(*Manager).initModule /opt/app-root/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138 github.com/grafana/dskit/modules.(*Manager).InitModuleServices /opt/app-root/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 github.com/grafana/loki/v3/pkg/loki.(*Loki).Run /opt/app-root/src/loki/pkg/loki/loki.go:549 main.main /opt/app-root/src/loki/cmd/loki/main.go:136 runtime.main /usr/lib/golang/src/runtime/proc.go:285 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1693 level=error ts=2026-03-05T18:58:17.823741028Z caller=log.go:223 msg="error running loki" err="init compactor: failed to init delete store: failed to get s3 object: operation error S3: GetObject, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region\nerror initialising module: compactor\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/opt/app-root/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/opt/app-root/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/opt/app-root/src/loki/pkg/loki/loki.go:549\nmain.main\n\t/opt/app-root/src/loki/cmd/loki/main.go:136\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693"
Ingester Logs:
level=error ts=2026-03-05T18:59:27.366794462Z caller=flush.go:262 component=ingester loop=25 org_id=infrastructure msg="failed to flush" retries=3 err="failed to flush chunks: store put chunk: operation error S3: PutObject, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region, num_chunks: 1, labels: {_time_shard_=\"1772715600_1772719200\", k8s_container_name=\"network-metrics-daemon\", k8s_namespace_name=\"openshift-multus\", k8s_node_name=\"ip-10-0-49-233.us-east-2.compute.internal\", k8s_pod_name=\"network-metrics-daemon-vzkkt\", kubernetes_container_name=\"network-metrics-daemon\", kubernetes_host=\"ip-10-0-49-233.us-east-2.compute.internal\", kubernetes_namespace_name=\"openshift-multus\", kubernetes_pod_name=\"network-metrics-daemon-vzkkt\", log_type=\"infrastructure\", openshift_log_type=\"infrastructure\"}"
LokiStack status
$ oc get pods -n openshift-logging -l app.kubernetes.io/instance=loki-74397 NAME READY STATUS RESTARTS AGE loki-74397-compactor-0 0/1 CrashLoopBackOff 14 (2m20s ago) 49m loki-74397-distributor-59f786956b-m5k6n 1/1 Running 0 60m loki-74397-gateway-9fd4f4847-7fs5d 2/2 Running 0 60m loki-74397-gateway-9fd4f4847-l5s5c 2/2 Running 0 59m loki-74397-index-gateway-0 1/1 Running 0 59m loki-74397-ingester-0 1/1 Running 0 61m loki-74397-querier-695cccd4cc-q6sb9 1/1 Running 0 61m loki-74397-query-frontend-6cf4fbf74c-mb4sh 1/1 Running 0 61m loki-74397-ruler-0 1/1 Running 0 60m
Credentials Request and Managed credentials:
$ oc get CredentialsRequest -n openshift-logging NAME AGE loki-74397 74m $ oc extract secret/loki-74397-managed-credentials -n openshift-logging credentials $ cat credentials [default] sts_regional_endpoints = regional role_arn = arn:aws:iam::<hidden>:role/loki-74397-1clqz5d8 web_identity_token_file = /var/run/secrets/storage/serviceaccount/token
Version-Release number of selected component (if applicable):
loki-operator.v6.5.0
How reproducible:
Always
Steps to Reproduce:
1) Deploy Loki Operator v6.5.0 on AWS STS cluster using role_arn
2) Create LokiStack secret required for AWS STS and provision LokiStack.
Actual results:
Compactor and Ingester throw Invalid Configuration: Missing Region in ingester storage/STS endpoint resolution
Expected results:
No error should be seen on LokiStack components in token-cco mode
Additional info:
No such issue seen on 6.4.2
