-
Bug
-
Resolution: Done
-
Critical
-
Logging 5.6.21, Logging 5.8.9, Logging 5.9.4, Logging 6.0.0
-
False
-
None
-
False
-
NEW
-
VERIFIED
-
Release Note Not Required
-
-
-
Log Storage - Sprint 256
-
Critical
Description of problem:
Creating lokistack with below yaml:
apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki namespace: openshift-logging spec: managementState: Managed rules: enabled: true namespaceSelector: matchLabels: openshift.io/cluster-monitoring: "true" selector: matchLabels: openshift.io/cluster-monitoring: "true" size: 1x.demo storage: secret: name: logging-storage-secret type: gcs storageClassName: standard-csi tenants: mode: openshift-logging
then check loki pods, many pods can't start and raise error:
$ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-9c5cb6478-bwsn6 1/1 Running 0 45m logging-loki-compactor-0 0/1 CrashLoopBackOff 13 (2m55s ago) 44m logging-loki-distributor-bf78bffc9-wzfb4 0/1 CrashLoopBackOff 13 (2m57s ago) 44m logging-loki-gateway-746f84dc98-k2j7m 2/2 Running 0 44m logging-loki-gateway-746f84dc98-xwb8w 2/2 Running 0 44m logging-loki-index-gateway-0 0/1 CrashLoopBackOff 13 (3m9s ago) 44m logging-loki-ingester-0 0/1 CrashLoopBackOff 13 (2m38s ago) 44m logging-loki-querier-6bf975c57b-q4xsv 0/1 CrashLoopBackOff 13 (3m2s ago) 44m logging-loki-query-frontend-7f964955d4-hs4xb 0/1 CrashLoopBackOff 13 (2m48s ago) 44m logging-loki-ruler-0 0/1 CrashLoopBackOff 13 (3m3s ago) 44m $ oc logs logging-loki-compactor-0 level=error ts=2024-07-08T01:19:06.952774524Z caller=main.go:70 msg="validating config" err="MULTIPLE CONFIG ERRORS FOUND, PLEASE READ CAREFULLY\nCONFIG ERROR: schema v13 is required to store Structured Metadata and use native OTLP ingestion, your schema version is v11. Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update to schema v13 or newer before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure\nCONFIG ERROR: `tsdb` index type is required to store Structured Metadata and use native OTLP ingestion, your index type is `boltdb-shipper` (defined in the `store` parameter of the schema_config). Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update the schema to use index type `tsdb` before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure"
When updating the spec.storage.schemas[0].version to `v13`, get below error:
# lokistacks.loki.grafana.com "logging-loki" was not valid: # * spec.storage.schemas[0].version: Unsupported value: "v13": supported values: "v11", "v12"
Version-Release number of selected component (if applicable):
openshift-logging/loki-operator-bundle/images/v5.8.9-19
openshift-logging/loki-rhel9-operator/images/v5.8.9-9
openshift-logging/logging-loki-rhel9/images/v3.1.0-3
How reproducible:
Always
Steps to Reproduce:
See above
Actual results:
Loki pods can't start.
Perviously, the loki version in logging 5.8 is v2.9.8, but in the new bundle, the version is changed to v3.1.0.
Expected results:
Loki pods should be ready without error
Additional info:
No issue when testing with openshift-logging-loki-rhel9-operator:v5.8.9-7, bundle image: openshift-logging-loki-operator-bundle:v5.8.9-17, loki image: openshift-logging-logging-loki-rhel9:v2.9.8-32
- links to
- mentioned on
(1 links to, 5 mentioned on)