When the galera controller starts a reconciliation, it assert the state of the attributes stored in the status object against the real state of the pods in the cluster.

If one of the pod was in the process of starting and crashed, it leaves outdated attributes in the status object. this gets cleaned up at the next reconciliation.

However it the crashed pod doesn't recreate its container on restart, but gets stuck in Pending phase, the galera controller crashes because it tries to access a pod's state that does not exist.

{{2024-08-09T15:15:19Z    INFO    Observed a panic in reconciler: runtime error: index out of range [0] with length 0     {"controller": "galera", "controllerGroup": "mariadb.openstack.org", "controllerKind": "Galera", "Galera":

, "namespace": "openstack", "name": "openstack-cell1", "reconcileID": "4a6c13f0-b927-4b59-99d4-3156a071503a"}}}
panic: runtime error: index out of range [0] with length 0 [recovered]
        panic: runtime error: index out of range [0] with length 0

goroutine 271 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.6/pkg/internal/controller/controller.go:116 +0x1fa
panic({0x1b1f6a0, 0xc002fa4be8})
        /usr/lib/golang/src/runtime/panic.go:884 +0x213
github.com/openstack-k8s-operators/mariadb-operator/controllers.assertPodsAttributesValidity(0xc000122630?, 0xc002fa2000, {0xc000273b00?, 0x3, 0xc002e3bd50?})
        /remote-source/controllers/galera_controller.go:285 +0x3cc
github.com/openstack-k8s-operators/mariadb-operator/controllers.(*GaleraReconciler).Reconcile(0xc0006dfe60, {0x1eca470, 0xc002f9d020}, {{{0xc0003b9586?, 0x5?}, {0xc0003b9590?, 0xc002335d48?})}}
        /remote-source/controllers/galera_controller.go:689 +0x352f
...