-
Bug
-
Resolution: Done
-
Normal
-
Logging 5.7.z, Logging 5.7.2
-
None
-
False
-
None
-
False
-
NEW
-
VERIFIED
-
Before this update, the Loki Operator crashed caused when a LokiStack custom resources defined with tenant limits only. With this update, the Loki Operator can process LokiStack custom resources without global limits resolves the issue.
-
Bug Fix
-
-
-
Log Storage - Sprint 237, Log Storage - Sprint 238
Description
Loki Controller manager crashes when LokiStack CR is created with limits per tenant.
This issue is not seen on 5.7.1
Error:
{"_ts":"2023-06-02T19:47:15.321825092Z","_level":"0","_component":"loki-operator","_message":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","LokiStack":{"name":"lokistack-dev","namespace":"openshift-logging"},"controller":"lokistack","controllerGroup":"loki.grafana.com","controllerKind":"LokiStack","name":"lokistack-dev","namespace":"openshift-logging","reconcileID":"c9979f30-0367-41ff-b3f6-ba753c507257"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1b3971c]
goroutine 742 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/remote-source/loki-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x1d2ff00, 0x344f050})
/usr/lib/golang/src/runtime/panic.go:884 +0x213
.....
CSV: loki-operator.v5.7.2
Steps to reproduce:
1) Deploy LO v5.7.2
2) Create LokiStack with per tenant limits.
LokiStack CR:
spec:
limits:
tenants:
application:
ingestion:
ingestionRate: 15
managementState: Managed
size: 1x.extra-small
storage:
schemas:
- effectiveDate: '2022-06-01'
version: v12
secret:
name: test
type: s3
storageClassName: standard-csi
tenants:
mode: openshift-logging
$ oc get pods -n openshift-operators-redhat NAME READY STATUS RESTARTS AGE elasticsearch-operator-54559968db-b2t2k 2/2 Running 0 110m loki-operator-controller-manager-6b44c9bd49-jdxjl 1/2 CrashLoopBackOff 5 (2m20s ago) 20m
How reproducible: Always
Expected Result: LokiStack should be running with no errors and overrides can be seen under runtime-config
Actual Result: Controller Crashes with runtime error
Additional Info:
Container image: registry.redhat.io/openshift-logging/loki-rhel8-operator@sha256:d1a921029b3079f68d052504742d774b287d203a81d5eafafd32185b4f823fa1
- is cloned by
-
-
- Closed
-
-
-
- Closed
-
- links to
-
RHBA-2023:6139
Logging Subsystem 5.8.0 - Red Hat OpenShift
- mentioned on
[LOG-4187] Controller crashes when only per tenant limits are defined in LokiStack CR
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Storage Midstream on branch openshift-logging-5.8-rhel-9_upstream_7c2a1904bacce268a74b3604a79a69f1:
Updated US source to: 30f7743 Merge pull request #151 from grafana/main
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Storage Midstream on branch openshift-logging-5.8-rhel-9_ upstream _7c2a1904bacce268a74b3604a79a69f1 : Updated US source to: 30f7743 Merge pull request #151 from grafana/main Update:
This issue does not occur if we define a global property as well. Below spec works fine and logs can be forwarded
limits: tenants: application: ingestion: ingestionRate: 15 global: ingestion: ingestionRate: 10
Moving ticket severity to Normal.
This issue has been addressed in the following products:
RHOL-5.8-RHEL-9
Via
RHBA-2023:6139 https://access.redhat.com/errata/RHBA-2023:6139