Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1401

eventrouter raising "Observed a panic: &runtime.TypeAssertionError"

XMLWordPrintable

    • False
    • False
    • NEW
    • NEW
    • Undefined
    • Logging (Core) - Sprint 201

      Description of problem:
      Using a pre-release image with fixes for #1859004 remains connected, but throws stack traces

      Version-Release number of selected component (if applicable):

      quay.io/vparfono/eventrouter:fixGoAway with https://github.com/openshift/eventrouter/pull/13 applied
      How reproducible:
      Always

      Steps to Reproduce:
      1. enable eventrouter

      Actual results:
      Occasional stack-traces are raised and eventrouter process crashes (will get restarted automatically):

      Time kubernetes.pod_name message
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg E0327 00:52:07.946730 1 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface*runtime._type)(0x15579a0), concrete*runtime._type)(0x1600100), asserted*runtime._type)(0x17379e0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg goroutine 120 [running]:
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg panic(0x1599380, 0xc0004ca3c0)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /usr/lib/golang/src/runtime/panic.go:969 +0x166
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:232
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run.func1()
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000ffdf60, 0x1982b20, 0xc00061e000, 0x1558901, 0xc00061c000)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000709760, 0x3b9aca00, 0x0, 0x1, 0xc00061c000)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg main.(*EventRouter).deleteEvent(0xc000418b80, 0x1600100, 0xc00069b600)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000709760)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:738 +0x95
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00037f4f0, 0xc00058e000)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1599380, 0xc0004ca3c0)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/eventrouter.go:197 +0xd6
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:746 +0x187
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.Until(...)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run(0xc0004bc500)
      March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x62
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg goroutine 120 [running]:
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic(0x1599380, 0xc0004ca3c0)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /usr/lib/golang/src/runtime/panic.go:969 +0x166
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.Until(...)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x62
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/eventrouter.go:197 +0xd6
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:232
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000709760, 0x3b9aca00, 0x0, 0x1, 0xc00061c000)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event [recovered]
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg main.(*EventRouter).deleteEvent(0xc000418b80, 0x1600100, 0xc00069b600)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run.func1()
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:746 +0x187
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000709760)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000ffdf60, 0x1982b20, 0xc00061e000, 0x1558901, 0xc00061c000)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run(0xc0004bc500)
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg /go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:738 +0x95
      March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00037f4f0, 0xc00058e000)
      March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271146 1 main.go:131] Starting shared Informer(s)
      March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271760 1 eventrouter.go:122] Starting EvenRouter
      March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271072 1 interfaces.go:37] Sink is [stdout]
      March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271176 1 main.go:118] Starting prometheus metrics.

      Expected results:
      No stack-traces and crashes of eventrouter.

      Additional info:

              vparfono Vitalii Parfonov
              jcantril@redhat.com Jeffrey Cantrill
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: