Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-4837

[Operator] Creating Backup fails and causes Operator to crash

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • RHDG 8.3 CD
    • RHDG 8.3 CD
    • Operator
    • None
    • False
    • False

      Creating Backup CR results in Operator crash due to invalid expectations in backup pod name.

      From Backup CR status:

      status: 
        phase: Failed
        reason: container infinispan is not valid for pod backup
      

      From the Operator logs:

      {"level":"error","ts":1631025065.40324,"logger":"backup-controller","msg":"Error encountered when cleaning up zero-capacity pod","error":"Unable to stop zero-capacity server: '': container infinispan is not valid for pod backup","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/root/go/src/github.com/infinispan/infinispan-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).cleanupResources\n\t/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:304\ngithub.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).Reconcile\n\t/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
      {"level":"info","ts":1631025065.4033072,"logger":"backup-controller","msg":"----- End Reconciling Backup","Request.Namespace":"datagrid","Request.Name":"backup"}
      E0907 14:31:05.403379       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      goroutine 2811 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic(0x15beaa0, 0x22c4e90)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
      panic(0x15beaa0, 0x22c4e90)
      	/usr/lib/golang/src/runtime/panic.go:969 +0x1b9
      github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).cleanupResources.func1(0x0, 0xc004575ac0)
      	/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:307 +0x22
      github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).cleanupResources(0xc000c05f90, 0x19689e0, 0xc001c68840, 0xc0009c52a0, 0x8, 0xc0009c5290, 0x6, 0x193b500, 0x0, 0x1930960, ...)
      	/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:313 +0x129
      github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).Reconcile(0xc000c05f90, 0xc0009c52a0, 0x8, 0xc0009c5290, 0x6, 0x658678b00, 0x0, 0x0, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:193 +0xbb9
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000aa2300, 0x16123a0, 0xc00093eb40, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000aa2300, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000aa2300)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b
      k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0044dc040)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0044dc040, 0x3b9aca00, 0x0, 0x1, 0xc00076a0c0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105
      k8s.io/apimachinery/pkg/util/wait.Until(0xc0044dc040, 0x3b9aca00, 0xc00076a0c0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x13f8b22]
      
      goroutine 2811 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
      panic(0x15beaa0, 0x22c4e90)
      	/usr/lib/golang/src/runtime/panic.go:969 +0x1b9
      github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).cleanupResources.func1(0x0, 0xc004575ac0)
      	/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:307 +0x22
      github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).cleanupResources(0xc000c05f90, 0x19689e0, 0xc001c68840, 0xc0009c52a0, 0x8, 0xc0009c5290, 0x6, 0x193b500, 0x0, 0x1930960, ...)
      	/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:313 +0x129
      github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity.(*Controller).Reconcile(0xc000c05f90, 0xc0009c52a0, 0x8, 0xc0009c5290, 0x6, 0x658678b00, 0x0, 0x0, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/pkg/controller/zerocapacity/zero_controller.go:193 +0xbb9
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000aa2300, 0x16123a0, 0xc00093eb40, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000aa2300, 0x0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000aa2300)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b
      k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0044dc040)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0044dc040, 0x3b9aca00, 0x0, 0x1, 0xc00076a0c0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105
      k8s.io/apimachinery/pkg/util/wait.Until(0xc0044dc040, 0x3b9aca00, 0xc00076a0c0)
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
      	/root/go/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d
      

            remerson@redhat.com Ryan Emerson
            pdrobek@redhat.com Pavel Drobek
            Pavel Drobek Pavel Drobek
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: