Uploaded image for project: 'Kogito'
  1. Kogito
  2. KOGITO-4993

Operator crash when data-index is created

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 1.6.0.Final
    • 1.5.0.Final
    • Operator
    • None
    • 2021 Week 16-18 (from Apr 19)

      OpenShift 4.7

      Installed Operator: Infinispan 2.1.3 (namespace), Strimzi 0.22.1 (namespace), Kogito 1.5.0 (cluster)

      created kogito-infinispan

      created kogito-kafka
      $ kogito install infra kogito-infinispan-infra --kind Infinispan --apiVersion infinispan.org/v1 --resource-name kogito-infinispan $ kogito install infra kogito-kafka-infra --kind Kafka --apiVersion kafka.strimzi.io/v1beta1 --resource-name kogito-kafka$ kogito install data-index --infra kogito-infinispan-infra --infra kogito-kafka-infra
      after the install of data-index the Kogito operator crashes.

      {"level":"info","ts":1619079043.5005922,"logger":"setup","msg":"The operator is running as cluster scoped. It will watch and manage resources in all namespaces","Env Var lookup":"WATCH_NAMESPACE"}

      {"level":"info","ts":1619079043.5005922,"logger":"setup","msg":"The operator is running as cluster scoped. It will watch and manage resources in all namespaces","Env Var lookup":"WATCH_NAMESPACE"}I0422 08:10:44.602150       1 request.go:655] Throttling request took 1.039664683s, request: GET:https://172.30.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s\{"level":"info","ts":1619079048.6483245,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}

      {"level":"info","ts":1619079048.6616101,"logger":"kogitoSupportingService-controller","msg":"Adding watched objects for KogitoSupportingService controller"}

      {"level":"info","ts":1619079048.6824713,"logger":"setup","msg":"starting manager"}I0422 08:10:48.682784       1 leaderelection.go:243] attempting to acquire leader lease openshift-operators/4662f1d5.kiegroup.org...{"level":"info","ts":1619079048.682852,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}[rbaumgar@rbaumgar ~]$ oc logs kogito-operator-controller-manager-84d5649749-zcg27 -n openshift-operators -c manager -f{"level":"info","ts":1619079043.5005922,"logger":"setup","msg":"The operator is running as cluster scoped. It will watch and manage resources in all namespaces","Env Var lookup":"WATCH_NAMESPACE"}I0422 08:10:44.602150       1 request.go:655] Throttling request took 1.039664683s, request: GET:https://172.30.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s\{"level":"info","ts":1619079048.6483245,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}

      {"level":"info","ts":1619079048.6616101,"logger":"kogitoSupportingService-controller","msg":"Adding watched objects for KogitoSupportingService controller"}

      {"level":"info","ts":1619079048.6824713,"logger":"setup","msg":"starting manager"}I0422 08:10:48.682784       1 leaderelection.go:243] attempting to acquire leader lease openshift-operators/4662f1d5.kiegroup.org...{"level":"info","ts":1619079048.682852,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}I0422 08:11:06.155021       1 leaderelection.go:253] successfully acquired lease openshift-operators/4662f1d5.kiegroup.org{"level":"info","ts":1619079066.1553063,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.1553729,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.155399,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.155472,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoBuild","controller":"kogitobuild","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.1554825,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.1555276,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoInfra","controller":"kogitoinfra","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.255851,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime"}

      {"level":"info","ts":1619079066.25588,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoBuild","controller":"kogitobuild","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.2560365,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice"}

      {"level":"info","ts":1619079066.2559073,"logger":"controller","msg":"Starting workers","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","worker count":1}

      {"level":"info","ts":1619079066.2560713,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.255877,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.256149,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoInfra","controller":"kogitoinfra","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.3563075,"logger":"controller","msg":"Starting workers","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","worker count":1}

      {"level":"info","ts":1619079066.3563743,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoBuild","controller":"kogitobuild","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.3564854,"logger":"KogitoSupportingService-finalizer","msg":"Reconciling for KogitoSupportingService finalizer","name":"data-index","namespace":"kogito-travel-agency"}

      {"level":"info","ts":1619079066.3565001,"logger":"KogitoSupportingService-finalizer","msg":"going to fetch deployed kogito supporting service","name":"data-index","namespace":"kogito-travel-agency"}

      {"level":"info","ts":1619079066.35692,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.3571324,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.4574752,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoBuild","controller":"kogitobuild"}

      {"level":"info","ts":1619079066.5575,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079066.5577042,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.0703018,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoInfra","controller":"kogitoinfra","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.071694,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoInfra","controller":"kogitoinfra"}

      {"level":"info","ts":1619079068.1595116,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.1616282,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.26026,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.2604394,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.2604814,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice"}

      {"level":"info","ts":1619079068.2605243,"logger":"controller","msg":"Starting workers","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoSupportingService","controller":"kogitosupportingservice","worker count":1}

      {"level":"info","ts":1619079068.2606285,"logger":"kogitoSupportingService-controller","msg":"Reconciling for KogitoSupportingService","name":"data-index","namespace":"kogito-travel-agency"}

      {"level":"info","ts":1619079068.2606452,"logger":"kogitoSupportingService-controller","msg":"going to fetch deployed kogito supporting service","name":"data-index","namespace":"kogito-travel-agency"}

      {"level":"info","ts":1619079068.2606847,"logger":"kogitoSupportingService-controller","msg":"Ensuring only single instance of supporting service exists","name":"data-index","namespace":"kogito-travel-agency"}

      {"level":"info","ts":1619079068.2612574,"logger":"kogitoSupportingService-controller","msg":"Reconciling for KogitoDataIndex","name":"data-index","namespace":"kogito-travel-agency","resource":"dataIndex","resource":"explainability","resource":"jobsService","resource":"mgmt-console","resource":"task-console","resource":"trusty-AI","resource":"trusty-AI"}

      {"level":"info","ts":1619079068.2614167,"logger":"kogitoSupportingService-controller","msg":"Updating status for Kogito Service","name":"data-index","namespace":"kogito-travel-agency","resource":"dataIndex","resource":"explainability","resource":"jobsService","resource":"mgmt-console","resource":"task-console","resource":"trusty-AI","resource":"trusty-AI"}

      {"level":"info","ts":1619079068.2619376,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.2620652,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","source":"kind source: /, Kind="}

      {"level":"info","ts":1619079068.26211,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime"}

      {"level":"info","ts":1619079068.2621493,"logger":"controller","msg":"Starting workers","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoRuntime","controller":"kogitoruntime","worker count":1}

      {"level":"info","ts":1619079068.272425,"logger":"controller","msg":"Starting workers","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoInfra","controller":"kogitoinfra","worker count":1}

      {"level":"info","ts":1619079068.2725835,"logger":"KogitoInfra-controller","msg":"Reconciling KogitoInfra","name":"kogito-infinispan-infra","namespace":"kogito-travel-agency"}

      {"level":"info","ts":1619079068.2729409,"logger":"controller","msg":"Starting workers","reconcilerGroup":"app.kiegroup.org","reconcilerKind":"KogitoBuild","controller":"kogitobuild","worker count":1}

      {"level":"info","ts":1619079068.4050748,"logger":"kogitoSupportingService-controller","msg":"Finished Kogito Service reconciliation","name":"data-index","namespace":"kogito-travel-agency","resource":"dataIndex","resource":"explainability","resource":"jobsService","resource":"mgmt-console","resource":"task-console","resource":"trusty-AI","resource":"trusty-AI"}E0422 08:11:08.405210       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)goroutine 721 [running]:k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1964ae0, 0x2d09fe0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/runtime/runtime.go:74 +0x92k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/runtime/runtime.go:48 +0x82panic(0x1964ae0, 0x2d09fe0) /usr/local/go/src/runtime/panic.go:969 +0x166github.com/kiegroup/kogito-operator/core/manager.(*kogitoInfraManager).IsKogitoInfraReady(0xc01087d7c0, 0xc000a2eb60, 0x14, 0xc000a2eba0, 0x12, 0x1, 0x0, 0x0) /workspace/core/manager/infra_manager.go:110 +0xb8github.com/kiegroup/kogito-operator/core/kogitoservice.(*serviceDeployer).checkInfraDependencies(0xc0010e60e0, 0x0, 0x0) /workspace/core/kogitoservice/deployer.go:224 +0x249github.com/kiegroup/kogito-operator/core/kogitoservice.(*serviceDeployer).Deploy(0xc0010e60e0, 0x0, 0x0, 0x0) /workspace/core/kogitoservice/deployer.go:142 +0x239github.com/kiegroup/kogito-operator/core/kogitosupportingservice.(*dataIndexSupportingServiceResource).Reconcile(0xc00fa41540, 0x1f1bec0, 0xc000cd78c0, 0x1e9de40) /workspace/core/kogitosupportingservice/dataindex_reconciler.go:68 +0x435github.com/kiegroup/kogito-operator/controllers.(*KogitoSupportingServiceReconciler).Reconcile(0xc0001d0040, 0xc000a2eb60, 0x14, 0xc000a00b60, 0xa, 0xc0110087b0, 0xc01087a120, 0xc0007981b8, 0xc0007981b0) /workspace/controllers/kogitosupportingservice_controller.go:94 +0x4a6sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000f27e0, 0x19d45c0, 0xc000ad47e0, 0xc000c20000) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244 +0x284sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000f27e0, 0x203000) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218 +0xaesigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0000f27e0) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197 +0x2bk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc011008780) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5fk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc011008780, 0x1e9e600, 0xc00f6ef4d0, 0xc00009c701, 0xc00009c7e0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xa3k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc011008780, 0x3b9aca00, 0x0, 0x10000c00010a101, 0xc00009c7e0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98k8s.io/apimachinery/pkg/util/wait.Until(0xc011008780, 0x3b9aca00, 0xc00009c7e0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90 +0x4dcreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:179 +0x42dpanic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1743288]
      goroutine 721 [running]:k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/runtime/runtime.go:55 +0x105panic(0x1964ae0, 0x2d09fe0) /usr/local/go/src/runtime/panic.go:969 +0x166github.com/kiegroup/kogito-operator/core/manager.(*kogitoInfraManager).IsKogitoInfraReady(0xc01087d7c0, 0xc000a2eb60, 0x14, 0xc000a2eba0, 0x12, 0x1, 0x0, 0x0) /workspace/core/manager/infra_manager.go:110 +0xb8github.com/kiegroup/kogito-operator/core/kogitoservice.(*serviceDeployer).checkInfraDependencies(0xc0010e60e0, 0x0, 0x0) /workspace/core/kogitoservice/deployer.go:224 +0x249github.com/kiegroup/kogito-operator/core/kogitoservice.(*serviceDeployer).Deploy(0xc0010e60e0, 0x0, 0x0, 0x0) /workspace/core/kogitoservice/deployer.go:142 +0x239github.com/kiegroup/kogito-operator/core/kogitosupportingservice.(*dataIndexSupportingServiceResource).Reconcile(0xc00fa41540, 0x1f1bec0, 0xc000cd78c0, 0x1e9de40) /workspace/core/kogitosupportingservice/dataindex_reconciler.go:68 +0x435github.com/kiegroup/kogito-operator/controllers.(*KogitoSupportingServiceReconciler).Reconcile(0xc0001d0040, 0xc000a2eb60, 0x14, 0xc000a00b60, 0xa, 0xc0110087b0, 0xc01087a120, 0xc0007981b8, 0xc0007981b0) /workspace/controllers/kogitosupportingservice_controller.go:94 +0x4a6sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000f27e0, 0x19d45c0, 0xc000ad47e0, 0xc000c20000) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244 +0x284sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000f27e0, 0x203000) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218 +0xaesigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0000f27e0) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197 +0x2bk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc011008780) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5fk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc011008780, 0x1e9e600, 0xc00f6ef4d0, 0xc00009c701, 0xc00009c7e0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xa3k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc011008780, 0x3b9aca00, 0x0, 0x10000c00010a101, 0xc00009c7e0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98k8s.io/apimachinery/pkg/util/wait.Until(0xc011008780, 0x3b9aca00, 0xc00009c7e0) /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90 +0x4dcreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:179 +0x42d

              ksuta Karel Suta
              rbaumgar Robert Baumgartner
              Karel Suta Karel Suta
              Karel Suta Karel Suta
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: