Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-37378

virt-controller panic: "invalid memory address or nil pointer dereference"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • CNV v4.16.0
    • CNV v4.12.8
    • CNV Virtualization
    • None
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • ---
    • ---
    • CNV Virtualization Sprint 249
    • High
    • No

      Description of problem:

      virt-controller pods are in a crash loop:

        containerStatuses:
  - containerID: cri-o://748d3be93da58acf3c753f0d455745a9480cc487c7409d880523549d797caa08
    image: registry.redhat.io/container-native-virtualization/virt-controller@sha256:fab26cd0c7b487af84c9186b7f9b50864ac0933bcb6890b288592031f805a186
    imageID: registry.redhat.io/container-native-virtualization/virt-controller@sha256:c7ef9a1f3dcd6ba257b3f6a5431cf8cce5809a4b76fa14523c65b45af995bc40
    lastState:
      terminated:
        containerID: cri-o://748d3be93da58acf3c753f0d455745a9480cc487c7409d880523549d797caa08
        exitCode: 2
        finishedAt: "2024-01-16T10:58:47Z"
        reason: Error
        startedAt: "2024-01-16T10:58:30Z"
    name: virt-controller
    ready: false
    restartCount: 6669
    started: false
    state:
      waiting:
        message: back-off 5m0s restarting failed container=virt-controller pod=virt-controller-7669fdcdd5-6cxxg_openshift-cnv(6e68a9ce-9d58-404e-94fd-9c5ca89f49a3)
        reason: CrashLoopBackOff

      The container is crashing with this panic:

      2024-01-16T10:58:47.655344909Z E0116 10:58:47.655257       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2024-01-16T10:58:47.655344909Z goroutine 1850 [running]:
2024-01-16T10:58:47.655344909Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1a51640?, 0x3147ba0})
2024-01-16T10:58:47.655344909Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x86
2024-01-16T10:58:47.655344909Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001be1bd0?})
2024-01-16T10:58:47.655344909Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
2024-01-16T10:58:47.655344909Z panic({0x1a51640, 0x3147ba0})
2024-01-16T10:58:47.655344909Z  /usr/lib/golang/src/runtime/panic.go:884 +0x212
2024-01-16T10:58:47.655344909Z k8s.io/apimachinery/pkg/util/intstr.(*IntOrString).IntValue(...)
2024-01-16T10:58:47.655344909Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/intstr/intstr.go:105
2024-01-16T10:58:47.655344909Z kubevirt.io/kubevirt/pkg/util/pdbs.IsPDBFromOldMigrationController(0xc003170ca8)
2024-01-16T10:58:47.655344909Z  /remote-source/app/pkg/util/pdbs/pdbs.go:39 +0x147
2024-01-16T10:58:47.655344909Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).sync(0xc0001d59c8?, {0xc00310b790, 0xd}, 0x1, 0xc0001d59c8, 0xc003170ca8)
2024-01-16T10:58:47.655344909Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:523 +0xc5
2024-01-16T10:58:47.655344909Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).execute(0xc001f7b080, {0xc00310b790, 0xd})
2024-01-16T10:58:47.655344909Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:392 +0x4ec
2024-01-16T10:58:47.655344909Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).Execute(0xc001f7b080)
2024-01-16T10:58:47.655344909Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:341 +0x108
2024-01-16T10:58:47.655344909Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).runWorker(0xc0032e8ea0?)
2024-01-16T10:58:47.655344909Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:331 +0x25
2024-01-16T10:58:47.655344909Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
2024-01-16T10:58:47.655344909Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
2024-01-16T10:58:47.655344909Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x212cd20, 0xc0015d92f0}, 0x1, 0xc001e1b8c0)
2024-01-16T10:58:47.655344909Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
2024-01-16T10:58:47.655344909Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0032e8fb0?, 0x3b9aca00, 0x0, 0x0?, 0x14dfae0?)
2024-01-16T10:58:47.655344909Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
2024-01-16T10:58:47.655344909Z k8s.io/apimachinery/pkg/util/wait.Until(0x9b704a?, 0xa03711101?, 0xc0032e8fa0?)
2024-01-16T10:58:47.655344909Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
2024-01-16T10:58:47.655344909Z created by kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).Run
2024-01-16T10:58:47.655344909Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:323 +0x36f
2024-01-16T10:58:47.658499723Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2024-01-16T10:58:47.658499723Z  panic: runtime error: invalid memory address or nil pointer dereference
2024-01-16T10:58:47.658499723Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17b9ee7]
2024-01-16T10:58:47.658515753Z 
2024-01-16T10:58:47.658515753Z goroutine 1850 [running]:
2024-01-16T10:58:47.658515753Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001be1bd0?})
2024-01-16T10:58:47.658515753Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd7
2024-01-16T10:58:47.658525276Z panic({0x1a51640, 0x3147ba0})
2024-01-16T10:58:47.658525276Z  /usr/lib/golang/src/runtime/panic.go:884 +0x212
2024-01-16T10:58:47.658532618Z k8s.io/apimachinery/pkg/util/intstr.(*IntOrString).IntValue(...)
2024-01-16T10:58:47.658532618Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/intstr/intstr.go:105
2024-01-16T10:58:47.658532618Z kubevirt.io/kubevirt/pkg/util/pdbs.IsPDBFromOldMigrationController(0xc003170ca8)
2024-01-16T10:58:47.658532618Z  /remote-source/app/pkg/util/pdbs/pdbs.go:39 +0x147
2024-01-16T10:58:47.658540523Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).sync(0xc0001d59c8?, {0xc00310b790, 0xd}, 0x1, 0xc0001d59c8, 0xc003170ca8)
2024-01-16T10:58:47.658554060Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:523 +0xc5
2024-01-16T10:58:47.658560907Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).execute(0xc001f7b080, {0xc00310b790, 0xd})
2024-01-16T10:58:47.658567789Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:392 +0x4ec
2024-01-16T10:58:47.658567789Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).Execute(0xc001f7b080)
2024-01-16T10:58:47.658579066Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:341 +0x108
2024-01-16T10:58:47.658579066Z kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).runWorker(0xc0032e8ea0?)
2024-01-16T10:58:47.658579066Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:331 +0x25
2024-01-16T10:58:47.658579066Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
2024-01-16T10:58:47.658586494Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
2024-01-16T10:58:47.658586494Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x212cd20, 0xc0015d92f0}, 0x1, 0xc001e1b8c0)
2024-01-16T10:58:47.658593663Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
2024-01-16T10:58:47.658600808Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0032e8fb0?, 0x3b9aca00, 0x0, 0x0?, 0x14dfae0?)
2024-01-16T10:58:47.658607610Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
2024-01-16T10:58:47.658607610Z k8s.io/apimachinery/pkg/util/wait.Until(0x9b704a?, 0xa03711101?, 0xc0032e8fa0?)
2024-01-16T10:58:47.658614997Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
2024-01-16T10:58:47.658622060Z created by kubevirt.io/kubevirt/pkg/virt-controller/watch/drain/disruptionbudget.(*DisruptionBudgetController).Run
2024-01-16T10:58:47.658622060Z  /remote-source/app/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go:323 +0x36f

       

      Version-Release number of selected component (if applicable):

      OCP Virt 4.12.8
      image: registry.redhat.io/container-native-virtualization/virt-controller@sha256:fab26cd0c7b487af84c9186b7f9b50864ac0933bcb6890b288592031f805a186

       

      How reproducible:

      Happens all the time in the customer environment. Not reproduced locally.

       

      Steps to Reproduce:

      1. ???
2.
3.

      Actual results:

       

      Expected results:

      virt-controller not crashing.

      Additional info:

       

              ffossemo@redhat.com Federico Fossemo
              rhn-support-jortialc Juan Orti
              Akriti gupta Akriti gupta
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: