example job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-jboss-fuse-camel-k-test-container-main-camel-k-ocp4.14-lp-interop-camel-k-interop-aws/1693503447845834752

: Undiagnosed panic detected in pod expand_less0s{  pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log.gz:E0821 06:29:09.418195       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)}

intermittent.  But seems to happen every few days or so:  $ podman run -it corbinu/alpine-w3m -dump -cols 200 "https://search.ci.openshift.org/?search=openshift-console-operator_console-operator.*invalid+memory+address+or+nil+pointer+dereference&maxAge=336h&context=1&type=junit&name=.*4.14.*&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job"
[openshift-console-op] [14d] [Links     ] [junit          ]
[Search]
Job:
[.*4.14.*            ] [                    ] [5                   ] [20971520            ] [job  ]
[ ] Wrap lines
periodic-ci-jboss-fuse-camel-k-test-container-main-camel-k-ocp4.14-lp-interop-camel-k-interop-aws (all) - 2 runs, 0% failed, 50% of runs match
#1693503447845834752                                                                                                                                      junit                                  4 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log.gz:E0821 06:29:09.418195       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-cpu-partitioning (all) - 54 runs, 7% failed, 25% of failures match = 2% impact
#1692398384507260928                                                                                                                                      junit                                  7 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-tppxt_console-operator.log.gz:E0818 05:14:35.489038       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-fips-serial (all) - 2 runs, 0% failed, 50% of runs match
#1692131892385550336                                                                                                                                      junit                                  8 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-lr54b_console-operator.log.gz:E0817 11:36:17.381961       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-gcp-ovn-upi (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1692132131523792896                                                                                                                                      junit                                  8 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-ztc7h_console-operator.log.gz:E0817 11:48:41.058274       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-shared-vpc-phz-techpreview (all) - 42 runs, 12% failed, 20% of failures match = 2% impact
#1691881846972878848                                                                                                                                      junit                                  8 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-ppmnm_console-operator.log.gz:E0816 19:12:20.716903       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-knative-serverless-operator-main-ocp4.14-lp-interop-operator-e2e-interop-aws-ocp414 (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1691835547812630528                                                                                                                                      junit                                  9 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-rm6fk_console-operator.log.gz:E0816 16:04:44.717384       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-gcp-ovn-rt (all) - 24 runs, 21% failed, 20% of failures match = 4% impact
#1691779459859877888                                                                                                                                      junit                                  9 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6df6498794-wqjk6_console-operator.log.gz:E0816 12:19:02.860700       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-cluster-etcd-operator-release-4.14-periodics-e2e-aws-etcd-recovery (all) - 14 runs, 100% failed, 7% of failures match = 7% impact
#1691303819960389632                                                                                                                                      junit                                  10 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-9f9fc5c5f-54xpb_console-operator_previous.log.gz:E0815 04:49:27.747727       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-proxy (all) - 34 runs, 24% failed, 13% of failures match = 3% impact
#1691023830476132352                                                                                                                                      junit                                  11 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6849dd7f6b-4r246_console-operator.log.gz:E0814 10:21:13.247983       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn (all) - 29 runs, 28% failed, 13% of failures match = 3% impact
#1690908928604377088                                                                                                                                      junit                                  11 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-bfcc5bfc7-qndlf_console-operator.log.gz:E0814 03:03:21.007675       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-csi (all) - 24 runs, 13% failed, 33% of failures match = 4% impact
#1690909013211877376                                                                                                                                      junit                                  11 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-bfcc5bfc7-6g886_console-operator.log.gz:E0814 02:43:39.815176       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-zones (all) - 2 runs, 0% failed, 50% of runs match
#1690604928210309120                                                                                                                                      junit                                  12 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-sfnjf_console-operator.log.gz:E0813 06:29:07.539267       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)


Found in 0.10% of runs (0.24% of failures) across 12026 total runs and 737 jobs (41.81% failed) in 1.057s - clear search | chart view - source code located on github

 

1.
2.
3.

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-jboss-fuse-camel-k-test-container-main-camel-k-ocp4.14-lp-interop-camel-k-interop-aws/1693503447845834752/artifacts/camel-k-interop-aws/gather-extra/artifacts/pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log has the stack trace:

E0821 06:29:09.418195       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 262 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x26aba20?, 0x475f720})
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x733dad?})
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x26aba20, 0x475f720})
	/usr/lib/golang/src/runtime/panic.go:884 +0x213
github.com/openshift/console-operator/pkg/console/controllers/util.IncludeNamesFilter.func1({0x27daba0?, 0xc00045a900?})
	/go/src/github.com/openshift/console-operator/pkg/console/controllers/util/util.go:34 +0x16f
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete({0xc0005e82d0?, {0x2f66350?, 0xc000d21b18?}}, {0x27daba0, 0xc00045a900})
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/client-go/tools/cache/controller.go:327 +0x46
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:978 +0xaf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x475d67?, {0x2f3e8e0, 0xc000da8b40}, 0x1, 0xc000df8060)
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc000fe8788?)
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000309710)
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:967 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x85