Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54422

Observed a panic for a whereabouts-controller pod after creating additionalNetworks

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.19
    • Networking / multus
    • None
    • +
    • Moderate
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      In 4.19, observed a panic for a whereabouts-controller pod after creating additionalNetworks

      Version-Release number of selected component (if applicable):

          4.19.0-0.nightly-2025-03-30-191359

      How reproducible:

      Always    

      Steps to Reproduce:

      1. After a OCP cluster up, oc edit networks.operator.openshift.io cluster
        additionalNetworks:
          - name: whereabouts-shim
            namespace: default
            type: Raw
            rawCNIConfig: |-
              {
                "cniVersion": "0.3.0",
                "type": "bridge",
                "name": "cnitest0",
                "ipam": {
                  "type": "whereabouts",
                  "subnet": "192.0.2.0/24"
                }
              }
      
      

      Actual results:

      $ oc get pod
      NAME                                           READY   STATUS    RESTARTS     AGE
      multus-6hhf8                                   1/1     Running   0            46m
      multus-additional-cni-plugins-4dd42            1/1     Running   0            46m
      multus-additional-cni-plugins-6b8gv            1/1     Running   0            46m
      multus-additional-cni-plugins-bq9f9            1/1     Running   0            30m
      multus-additional-cni-plugins-nn9qk            1/1     Running   0            46m
      multus-additional-cni-plugins-t8xd2            1/1     Running   0            30m
      multus-admission-controller-674d95ff59-k6vpq   2/2     Running   0            43m
      multus-admission-controller-674d95ff59-s6flh   2/2     Running   0            42m
      multus-cbqd6                                   1/1     Running   0            30m
      multus-rhbmg                                   1/1     Running   0            46m
      multus-sjlkv                                   1/1     Running   0            30m
      multus-trrbf                                   1/1     Running   0            46m
      network-metrics-daemon-5vb4k                   2/2     Running   0            30m
      network-metrics-daemon-knxt6                   2/2     Running   0            46m
      network-metrics-daemon-qfw76                   2/2     Running   0            30m
      network-metrics-daemon-sklp9                   2/2     Running   0            46m
      network-metrics-daemon-v8nkr                   2/2     Running   0            46m
      whereabouts-controller-688f5c65b4-gjmj7        0/1     Error     1 (2s ago)   15s
      whereabouts-reconciler-bvfkh                   1/1     Running   0            15s
      whereabouts-reconciler-fstd5                   1/1     Running   0            15s
      whereabouts-reconciler-qwsf4                   1/1     Running   0            15s
      whereabouts-reconciler-t22v7                   1/1     Running   0            15s
      whereabouts-reconciler-x66gw                   1/1     Running   0            15s
      $ oc logs whereabouts-controller-688f5c65b4-gjmj7
      W0331 13:56:04.562462       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
      I0331 13:56:04.563283       1 controller.go:130] "Setting up event handlers"
      I0331 13:56:04.563335       1 controller.go:209] "Starting node-slice controller"
      I0331 13:56:04.563345       1 controller.go:212] "Waiting for informer caches to sync"
      I0331 13:56:04.659965       1 controller.go:155] handling network attachment definition event
      I0331 13:56:04.660033       1 controller.go:171] default/whereabouts-shim
      I0331 13:56:04.662695       1 controller.go:183] handling requeueNADs
      I0331 13:56:04.662713       1 controller.go:183] handling requeueNADs
      I0331 13:56:04.662725       1 controller.go:183] handling requeueNADs
      I0331 13:56:04.662732       1 controller.go:183] handling requeueNADs
      I0331 13:56:04.662741       1 controller.go:183] handling requeueNADs
      I0331 13:56:04.663431       1 controller.go:224] "Starting workers" count=1
      I0331 13:56:04.663446       1 controller.go:230] "Started workers"
      2025-03-31T13:56:04Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
      2025-03-31T13:56:04Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
      E0331 13:56:04.664074       1 panic.go:115] "Observed a panic" panic="runtime error: index out of range [0] with length 0" panicGoValue="runtime.boundsError{x:0, y:0, signed:true, code:0x0}" stacktrace=<
          goroutine 63 [running]:
          k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1fc4a18, 0x3184aa0}, {0x1bf6f40, 0xc00049a960})
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc
          k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x1fc4a18, 0x3184aa0}, {0x1bf6f40, 0xc00049a960}, {0x3184aa0, 0x0, 0x10000000043ff85?})
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e
          k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000134540?})
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
          panic({0x1bf6f40?, 0xc00049a960?})
              /usr/lib/golang/src/runtime/panic.go:785 +0x132
          github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.checkIpamConfMatch(...)
              /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:526
          github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).checkForMultiNadMismatch(0xc0001e70a0, {0xc00049a308, 0x10}, {0xc00049a300?, 0x0?})
              /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:517 +0x255
          github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).syncHandler(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30}, {0xc00049a300, 0x18})
              /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:304 +0x16a
          github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem.func1(0xc0001e70a0, {0x1fc4ba0?, 0xc000585b30?}, {{0x1fc82e0?, 0xc0000aad80?}, 0xc0001ede60?}, {0xc00049a300, 0x18})
              /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:272 +0xd8
          github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30})
              /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:282 +0x85
          github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).runWorker(...)
              /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:241
          k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x1f
          k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
          k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00044ff70, {0x1fa58e0, 0xc000230cc0}, 0x1, 0xc0000b6690)
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
          k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0001edf70, 0x3b9aca00, 0x0, 0x1, 0xc0000b6690)
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
          k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x1fc4ba0, 0xc000585b30}, 0xc0000548c0, 0x3b9aca00, 0x0, 0x1)
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x87
          k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
              /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170
          created by github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).Run in goroutine 1
              /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:227 +0x3ae
       >
      panic: runtime error: index out of range [0] with length 0 [recovered]
          panic: runtime error: index out of range [0] with length 0goroutine 63 [running]:
      k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x1fc4a18, 0x3184aa0}, {0x1bf6f40, 0xc00049a960}, {0x3184aa0, 0x0, 0x10000000043ff85?})
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000134540?})
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
      panic({0x1bf6f40?, 0xc00049a960?})
          /usr/lib/golang/src/runtime/panic.go:785 +0x132
      github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.checkIpamConfMatch(...)
          /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:526
      github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).checkForMultiNadMismatch(0xc0001e70a0, {0xc00049a308, 0x10}, {0xc00049a300?, 0x0?})
          /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:517 +0x255
      github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).syncHandler(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30}, {0xc00049a300, 0x18})
          /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:304 +0x16a
      github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem.func1(0xc0001e70a0, {0x1fc4ba0?, 0xc000585b30?}, {{0x1fc82e0?, 0xc0000aad80?}, 0xc0001ede60?}, {0xc00049a300, 0x18})
          /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:272 +0xd8
      github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30})
          /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:282 +0x85
      github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).runWorker(...)
          /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:241
      k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x1f
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00044ff70, {0x1fa58e0, 0xc000230cc0}, 0x1, 0xc0000b6690)
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0001edf70, 0x3b9aca00, 0x0, 0x1, 0xc0000b6690)
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
      k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x1fc4ba0, 0xc000585b30}, 0xc0000548c0, 0x3b9aca00, 0x0, 0x1)
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x87
      k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
          /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170
      created by github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).Run in goroutine 1
          /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:227 +0x3ae    $ oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.19.0-0.nightly-2025-03-30-191359   True        False         34m     Cluster version is 4.19.0-0.nightly-2025-03-30-191359
      

      Expected results:

        whereabouts-controller pod is created and in Running state  

      Additional info:

          

              bpickard@redhat.com Ben Pickard
              weliang1@redhat.com Weibin Liang
              Weibin Liang Weibin Liang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: