Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5778

[Logging 5.8.z]Loki pods raise `CONFIG ERROR: schema v13 is required to store Structured Metadata and use native OTLP ingestion` however v13 is not supported in lokistack

XMLWordPrintable

    • Log Storage - Sprint 256
    • Critical

      Description of problem:

      Creating lokistack with below yaml:

      apiVersion: loki.grafana.com/v1
      kind: LokiStack
      metadata:
        name: logging-loki
        namespace: openshift-logging
      spec:
        managementState: Managed
        rules:
          enabled: true
          namespaceSelector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"
          selector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"
        size: 1x.demo
        storage:
          secret:
            name: logging-storage-secret
            type: gcs
        storageClassName: standard-csi
        tenants:
          mode: openshift-logging 

      then check loki pods, many pods can't start and raise error:

      $ oc get pod
      NAME                                           READY   STATUS             RESTARTS         AGE
      cluster-logging-operator-9c5cb6478-bwsn6       1/1     Running            0                45m
      logging-loki-compactor-0                       0/1     CrashLoopBackOff   13 (2m55s ago)   44m
      logging-loki-distributor-bf78bffc9-wzfb4       0/1     CrashLoopBackOff   13 (2m57s ago)   44m
      logging-loki-gateway-746f84dc98-k2j7m          2/2     Running            0                44m
      logging-loki-gateway-746f84dc98-xwb8w          2/2     Running            0                44m
      logging-loki-index-gateway-0                   0/1     CrashLoopBackOff   13 (3m9s ago)    44m
      logging-loki-ingester-0                        0/1     CrashLoopBackOff   13 (2m38s ago)   44m
      logging-loki-querier-6bf975c57b-q4xsv          0/1     CrashLoopBackOff   13 (3m2s ago)    44m
      logging-loki-query-frontend-7f964955d4-hs4xb   0/1     CrashLoopBackOff   13 (2m48s ago)   44m
      logging-loki-ruler-0                           0/1     CrashLoopBackOff   13 (3m3s ago)    44m 
      
      $ oc logs logging-loki-compactor-0 
      level=error ts=2024-07-08T01:19:06.952774524Z caller=main.go:70 msg="validating config" err="MULTIPLE CONFIG ERRORS FOUND, PLEASE READ CAREFULLY\nCONFIG ERROR: schema v13 is required to store Structured Metadata and use native OTLP ingestion, your schema version is v11. Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update to schema v13 or newer before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure\nCONFIG ERROR: `tsdb` index type is required to store Structured Metadata and use native OTLP ingestion, your index type is `boltdb-shipper` (defined in the `store` parameter of the schema_config). Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update the schema to use index type `tsdb` before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure"

      When updating the spec.storage.schemas[0].version to `v13`, get below error:

       # lokistacks.loki.grafana.com "logging-loki" was not valid:
       # * spec.storage.schemas[0].version: Unsupported value: "v13": supported values: "v11", "v12" 

      Version-Release number of selected component (if applicable):

      openshift-logging/loki-operator-bundle/images/v5.8.9-19

      openshift-logging/loki-rhel9-operator/images/v5.8.9-9

      openshift-logging/logging-loki-rhel9/images/v3.1.0-3

      How reproducible:

      Always

      Steps to Reproduce:

      See above

      Actual results:

      Loki pods can't start.

      Perviously, the loki version in logging 5.8 is v2.9.8, but in the new bundle, the version is changed to v3.1.0.

      Expected results:

      Loki pods should be ready without error

      Additional info:

      No issue when testing with openshift-logging-loki-rhel9-operator:v5.8.9-7, bundle image: openshift-logging-loki-operator-bundle:v5.8.9-17, loki image: openshift-logging-logging-loki-rhel9:v2.9.8-32

              ptsiraki@redhat.com Periklis Tsirakidis
              qitang@redhat.com Qiaoling Tang
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: