-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Logging 5.8.0, Logging 5.9.0
-
False
-
None
-
False
-
NEW
-
NEW
-
Bug Fix
-
-
-
Log Storage - Sprint 252
Description of problem:
When LokiStack is configured to use Alibaba cloud storage bucket, Loki components are down with CrashLoop error.
Error:
bucket name len is between [3-63],now is 0 error creating object client github.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod /remote-source/loki/app/pkg/storage/store.go:202 github.com/grafana/loki/pkg/storage.(*store).init /remote-source/loki/app/pkg/storage/store.go:164 github.com/grafana/loki/pkg/storage.NewStore /remote-source/loki/app/pkg/storage/store.go:155 github.com/grafana/loki/pkg/loki.(*Loki).initStore /remote-source/loki/app/pkg/loki/modules.go:689 github.com/grafana/dskit/modules.(*Manager).initModule ......
$ oc get pods -n openshift-logging
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-6d584c87b5-w5xqn 1/1 Running 0 9m15s
logging-loki-compactor-0 0/1 CrashLoopBackOff 3 (22s ago) 97s
logging-loki-distributor-5db7d759f6-fx82h 1/1 Running 0 97s
logging-loki-gateway-647f74cd56-85mkr 2/2 Running 0 96s
logging-loki-gateway-647f74cd56-hfxt9 2/2 Running 0 96s
logging-loki-index-gateway-0 0/1 CrashLoopBackOff 3 (31s ago) 96s
logging-loki-ingester-0 0/1 CrashLoopBackOff 3 (41s ago) 97s
logging-loki-querier-6cd8867fc6-7ngqs 0/1 CrashLoopBackOff 4 (11s ago) 97s
logging-loki-query-frontend-56b99dbf59-r8g7q 1/1 Running 0 97s
logging-loki-ruler-0 0/1 CrashLoopBackOff 3 (39s ago) 96s
Storage secret:
oc create secret generic logging-loki-bucket-creds \ --from-literal=bucket="logging-loki-kbharti" \ --from-literal=endpoint="oss-us-east-1.aliyuncs.com" \ --from-literal=access_key_id="<hidden>" \ --from-literal=secret_access_key="<hidden>"
LokiStack spec:
spec: managementState: Managed rules: enabled: true namespaceSelector: matchLabels: openshift.io/cluster-monitoring: 'true' selector: matchLabels: openshift.io/cluster-monitoring: 'true' size: 1x.demo storage: schemas: - effectiveDate: '2023-10-15' version: v13 secret: name: logging-loki-bucket-creds type: alibabacloud storageClassName: alicloud-disk tenants: mode: openshift-logging
config.yaml
..
common:
storage:
alibabacloud:
bucket: logging-loki-kbharti
endpoint: oss-us-east-1.aliyuncs.com
access_key_id: ${ALIBABA_CLOUD_ACCESS_KEY_ID}
secret_access_key: ${ALIBABA_CLOUD_ACCESS_KEY_SECRET}
compactor_grpc_address: logging-loki-compactor-grpc.openshift-logging.svc.cluster.local:9095
...
How reproducible: Always
Version: Logging 5.9, Logging 5.8 etc.
Expected Result: LokiStack components should be up and running without error
Actual Result: LokiStack components are down with mentioned error
Additional info/Workaround:
Mark LokiStack as unmanaged and move storage config under storage_config in config.yaml
...
client_ca_file: /var/run/ca/service-ca.crt
log_level: info
storage_config:
alibabacloud:
bucket: logging-loki-kbharti
endpoint: oss-us-east-1.aliyuncs.com
access_key_id: <hidden>
secret_access_key: <hidden>
tsdb_shipper:
...
Restart the loki components
$ oc get pods -n openshift-logging
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-6d584c87b5-w5xqn 1/1 Running 0 16m
logging-loki-compactor-0 1/1 Running 0 2m28s
logging-loki-distributor-5db7d759f6-fx82h 1/1 Running 0 8m22s
logging-loki-gateway-647f74cd56-85mkr 2/2 Running 0 8m21s
logging-loki-gateway-647f74cd56-hfxt9 2/2 Running 0 8m21s
logging-loki-index-gateway-0 1/1 Running 0 2m21s
logging-loki-ingester-0 1/1 Running 6 (5m8s ago) 8m22s
logging-loki-querier-6cd8867fc6-cc56x 1/1 Running 0 2m18s
logging-loki-query-frontend-56b99dbf59-r8g7q 1/1 Running 0 8m22s
logging-loki-ruler-0 1/1 Running 0 2m13s