Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15244

Undiagnosed panic detected in pods/openshift-ovn-kubernetes_ovnkube-master-zc4bf_ovnkube-master_previous

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 4.14
    • None
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None

    Description

      This was noticed when bringing in k8s 1.27.3:

      https://github.com/openshift/kubernetes/pull/1609

      the exact job is:

      https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_kubernetes/1609/pull-ci-openshift-kubernetes-master-e2e-aws-ovn-serial/1671272159323885568

      the panic is:

      E0620 22:33:59.159702       1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x1dd8820), concrete:(*runtime._type)(0x1f01740), asserted:(*runtime._type)(0x20f42e0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod)
      goroutine 666 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1e54460?, 0xc001067b00})
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00005e5e0?})
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
      panic({0x1e54460, 0xc001067b00})
      	/usr/lib/golang/src/runtime/panic.go:884 +0x213
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).processNextPodWorkItem(0xc0004f88f0, 0xc0006feb20)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:466 +0x586
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).runPodWorker(...)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:450
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).Run.func2.1()
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:216 +0x22
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x24706a0, 0xc000c9fe60}, 0x1, 0xc0004ed800)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).Run.func2(0xc0006724e0)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:215 +0xcf
      created by github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).Run
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:212 +0x51a
      panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod [recovered]
      	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod
      
      goroutine 666 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00005e5e0?})
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
      panic({0x1e54460, 0xc001067b00})
      	/usr/lib/golang/src/runtime/panic.go:884 +0x213
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).processNextPodWorkItem(0xc0004f88f0, 0xc0006feb20)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:466 +0x586
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).runPodWorker(...)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:450
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).Run.func2.1()
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:216 +0x22
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x24706a0, 0xc000c9fe60}, 0x1, 0xc0004ed800)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).Run.func2(0xc0006724e0)
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:215 +0xcf
      created by github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute.(*ExternalGatewayMasterController).Run
      	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/controller/apbroute/master_controller.go:212 +0x51a
      

      it only happened once, but it seems like there's a wrong handling of terminating pods.

      Attachments

        Issue Links

          Activity

            People

              jgil@redhat.com Jordi Gil
              maszulik1@redhat.com Maciej Szulik
              Anurag Saxena Anurag Saxena
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: