Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4198

[release-5.7] Controller crashes when only per tenant limits are defined in LokiStack CR

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Before this update, the Loki Operator crashed caused when a LokiStack custom resources defined with tenant limits only. With this update, the Loki Operator can process LokiStack custom resources without global limits resolves the issue.
    • Bug Fix
    • Log Storage - Sprint 237, Log Storage - Sprint 238

      Description
      Loki Controller manager crashes when LokiStack CR is created with limits per tenant.

      This issue is not seen on 5.7.1

      Error:

      {"_ts":"2023-06-02T19:47:15.321825092Z","_level":"0","_component":"loki-operator","_message":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","LokiStack":{"name":"lokistack-dev","namespace":"openshift-logging"},"controller":"lokistack","controllerGroup":"loki.grafana.com","controllerKind":"LokiStack","name":"lokistack-dev","namespace":"openshift-logging","reconcileID":"c9979f30-0367-41ff-b3f6-ba753c507257"}
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1b3971c]
      goroutine 742 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
      /remote-source/loki-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa
      panic({0x1d2ff00, 0x344f050})
      /usr/lib/golang/src/runtime/panic.go:884 +0x213
      ..... 
      

      CSV: loki-operator.v5.7.2

      Steps to reproduce:
      1) Deploy LO v5.7.2
      2) Create LokiStack with per tenant limits.

      LokiStack CR:

      spec:
        limits:
          tenants:
            application:
              ingestion:
                ingestionRate: 15
        managementState: Managed
        size: 1x.extra-small
        storage:
          schemas:
            - effectiveDate: '2022-06-01'
              version: v12
          secret:
            name: test
            type: s3
        storageClassName: standard-csi
        tenants:
          mode: openshift-logging
      $ oc get pods -n openshift-operators-redhat
      NAME                                                READY   STATUS             RESTARTS        AGE
      elasticsearch-operator-54559968db-b2t2k             2/2     Running            0               110m
      loki-operator-controller-manager-6b44c9bd49-jdxjl   1/2     CrashLoopBackOff   5 (2m20s ago)   20m

      How reproducible: Always

      Expected Result: LokiStack should be running with no errors and overrides can be seen under runtime-config

      Actual Result: Controller Crashes with runtime error

      Additional Info: 
      Container image: registry.redhat.io/openshift-logging/loki-rhel8-operator@sha256:d1a921029b3079f68d052504742d774b287d203a81d5eafafd32185b4f823fa1

            ptsiraki@redhat.com Periklis Tsirakidis
            rhn-support-kbharti Kabir Bharti
            Kabir Bharti Kabir Bharti
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: