-
Bug
-
Resolution: Done
-
Normal
-
None
-
False
-
False
-
NEW
-
NEW
-
Undefined
-
-
Logging (Core) - Sprint 204
Description of problem:
Using a pre-release image with fixes for #1859004 remains connected, but throws stack traces
Version-Release number of selected component (if applicable):
quay.io/vparfono/eventrouter:fixGoAway with https://github.com/openshift/eventrouter/pull/13 applied
How reproducible:
Always
Steps to Reproduce:
1. enable eventrouter
Actual results:
Occasional stack-traces are raised and eventrouter process crashes (will get restarted automatically):
Time kubernetes.pod_name message
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg E0327 00:52:07.946730 1 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface*runtime._type)(0x15579a0), concrete*runtime._type)(0x1600100), asserted*runtime._type)(0x17379e0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg goroutine 120 [running]:
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg panic(0x1599380, 0xc0004ca3c0)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /usr/lib/golang/src/runtime/panic.go:969 +0x166
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:232
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run.func1()
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000ffdf60, 0x1982b20, 0xc00061e000, 0x1558901, 0xc00061c000)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000709760, 0x3b9aca00, 0x0, 0x1, 0xc00061c000)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg main.(*EventRouter).deleteEvent(0xc000418b80, 0x1600100, 0xc00069b600)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000709760)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:738 +0x95
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00037f4f0, 0xc00058e000)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1599380, 0xc0004ca3c0)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/eventrouter.go:197 +0xd6
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:746 +0x187
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.Until(...)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run(0xc0004bc500)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x62
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg goroutine 120 [running]:
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic(0x1599380, 0xc0004ca3c0)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /usr/lib/golang/src/runtime/panic.go:969 +0x166
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.Until(...)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x62
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/eventrouter.go:197 +0xd6
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:232
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000709760, 0x3b9aca00, 0x0, 0x1, 0xc00061c000)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event [recovered]
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg main.(*EventRouter).deleteEvent(0xc000418b80, 0x1600100, 0xc00069b600)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run.func1()
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:746 +0x187
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000709760)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000ffdf60, 0x1982b20, 0xc00061e000, 0x1558901, 0xc00061c000)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run(0xc0004bc500)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:738 +0x95
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00037f4f0, 0xc00058e000)
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271146 1 main.go:131] Starting shared Informer(s)
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271760 1 eventrouter.go:122] Starting EvenRouter
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271072 1 interfaces.go:37] Sink is [stdout]
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271176 1 main.go:118] Starting prometheus metrics.
Expected results:
No stack-traces and crashes of eventrouter.
Additional info:
- clones
-
LOG-1401 eventrouter raising "Observed a panic: &runtime.TypeAssertionError"
- Closed