Loading...

XML

Word

Printable

Type: Bug
Resolution: Done-Errata
Priority: Major
Fix Version/s: 4.19.0
Affects Version/s: 4.19
Component/s: Networking / multus
Labels:
None

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
Moderate
Regression:
No

Target Backport Versions:
None
Target Version:

4.19
Release Blocker:
Proposed
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Test Coverage:

+

Release Note Status:
Done
Release Note Type:
Release Note Not Required
Release Note Text:
N/A

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Description of problem:

In 4.19, observed a panic for a whereabouts-controller pod after creating additionalNetworks

Version-Release number of selected component (if applicable):

    4.19.0-0.nightly-2025-03-30-191359

How reproducible:

Always

Steps to Reproduce:

1. After a OCP cluster up, oc edit networks.operator.openshift.io cluster
  additionalNetworks:
    - name: whereabouts-shim
      namespace: default
      type: Raw
      rawCNIConfig: |-
        {
          "cniVersion": "0.3.0",
          "type": "bridge",
          "name": "cnitest0",
          "ipam": {
            "type": "whereabouts",
            "subnet": "192.0.2.0/24"
          }
        }

Actual results:

$ oc get pod
NAME                                           READY   STATUS    RESTARTS     AGE
multus-6hhf8                                   1/1     Running   0            46m
multus-additional-cni-plugins-4dd42            1/1     Running   0            46m
multus-additional-cni-plugins-6b8gv            1/1     Running   0            46m
multus-additional-cni-plugins-bq9f9            1/1     Running   0            30m
multus-additional-cni-plugins-nn9qk            1/1     Running   0            46m
multus-additional-cni-plugins-t8xd2            1/1     Running   0            30m
multus-admission-controller-674d95ff59-k6vpq   2/2     Running   0            43m
multus-admission-controller-674d95ff59-s6flh   2/2     Running   0            42m
multus-cbqd6                                   1/1     Running   0            30m
multus-rhbmg                                   1/1     Running   0            46m
multus-sjlkv                                   1/1     Running   0            30m
multus-trrbf                                   1/1     Running   0            46m
network-metrics-daemon-5vb4k                   2/2     Running   0            30m
network-metrics-daemon-knxt6                   2/2     Running   0            46m
network-metrics-daemon-qfw76                   2/2     Running   0            30m
network-metrics-daemon-sklp9                   2/2     Running   0            46m
network-metrics-daemon-v8nkr                   2/2     Running   0            46m
whereabouts-controller-688f5c65b4-gjmj7        0/1     Error     1 (2s ago)   15s
whereabouts-reconciler-bvfkh                   1/1     Running   0            15s
whereabouts-reconciler-fstd5                   1/1     Running   0            15s
whereabouts-reconciler-qwsf4                   1/1     Running   0            15s
whereabouts-reconciler-t22v7                   1/1     Running   0            15s
whereabouts-reconciler-x66gw                   1/1     Running   0            15s
$ oc logs whereabouts-controller-688f5c65b4-gjmj7
W0331 13:56:04.562462       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0331 13:56:04.563283       1 controller.go:130] "Setting up event handlers"
I0331 13:56:04.563335       1 controller.go:209] "Starting node-slice controller"
I0331 13:56:04.563345       1 controller.go:212] "Waiting for informer caches to sync"
I0331 13:56:04.659965       1 controller.go:155] handling network attachment definition event
I0331 13:56:04.660033       1 controller.go:171] default/whereabouts-shim
I0331 13:56:04.662695       1 controller.go:183] handling requeueNADs
I0331 13:56:04.662713       1 controller.go:183] handling requeueNADs
I0331 13:56:04.662725       1 controller.go:183] handling requeueNADs
I0331 13:56:04.662732       1 controller.go:183] handling requeueNADs
I0331 13:56:04.662741       1 controller.go:183] handling requeueNADs
I0331 13:56:04.663431       1 controller.go:224] "Starting workers" count=1
I0331 13:56:04.663446       1 controller.go:230] "Started workers"
2025-03-31T13:56:04Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2025-03-31T13:56:04Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
E0331 13:56:04.664074       1 panic.go:115] "Observed a panic" panic="runtime error: index out of range [0] with length 0" panicGoValue="runtime.boundsError{x:0, y:0, signed:true, code:0x0}" stacktrace=<
    goroutine 63 [running]:
    k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1fc4a18, 0x3184aa0}, {0x1bf6f40, 0xc00049a960})
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc
    k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x1fc4a18, 0x3184aa0}, {0x1bf6f40, 0xc00049a960}, {0x3184aa0, 0x0, 0x10000000043ff85?})
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e
    k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000134540?})
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
    panic({0x1bf6f40?, 0xc00049a960?})
        /usr/lib/golang/src/runtime/panic.go:785 +0x132
    github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.checkIpamConfMatch(...)
        /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:526
    github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).checkForMultiNadMismatch(0xc0001e70a0, {0xc00049a308, 0x10}, {0xc00049a300?, 0x0?})
        /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:517 +0x255
    github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).syncHandler(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30}, {0xc00049a300, 0x18})
        /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:304 +0x16a
    github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem.func1(0xc0001e70a0, {0x1fc4ba0?, 0xc000585b30?}, {{0x1fc82e0?, 0xc0000aad80?}, 0xc0001ede60?}, {0xc00049a300, 0x18})
        /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:272 +0xd8
    github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30})
        /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:282 +0x85
    github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).runWorker(...)
        /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:241
    k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x1f
    k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
    k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00044ff70, {0x1fa58e0, 0xc000230cc0}, 0x1, 0xc0000b6690)
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
    k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0001edf70, 0x3b9aca00, 0x0, 0x1, 0xc0000b6690)
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
    k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x1fc4ba0, 0xc000585b30}, 0xc0000548c0, 0x3b9aca00, 0x0, 0x1)
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x87
    k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
        /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170
    created by github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).Run in goroutine 1
        /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:227 +0x3ae
 >
panic: runtime error: index out of range [0] with length 0 [recovered]
    panic: runtime error: index out of range [0] with length 0goroutine 63 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x1fc4a18, 0x3184aa0}, {0x1bf6f40, 0xc00049a960}, {0x3184aa0, 0x0, 0x10000000043ff85?})
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000134540?})
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
panic({0x1bf6f40?, 0xc00049a960?})
    /usr/lib/golang/src/runtime/panic.go:785 +0x132
github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.checkIpamConfMatch(...)
    /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:526
github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).checkForMultiNadMismatch(0xc0001e70a0, {0xc00049a308, 0x10}, {0xc00049a300?, 0x0?})
    /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:517 +0x255
github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).syncHandler(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30}, {0xc00049a300, 0x18})
    /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:304 +0x16a
github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem.func1(0xc0001e70a0, {0x1fc4ba0?, 0xc000585b30?}, {{0x1fc82e0?, 0xc0000aad80?}, 0xc0001ede60?}, {0xc00049a300, 0x18})
    /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:272 +0xd8
github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).processNextWorkItem(0xc0001e70a0, {0x1fc4ba0, 0xc000585b30})
    /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:282 +0x85
github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).runWorker(...)
    /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:241
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x1f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00044ff70, {0x1fa58e0, 0xc000230cc0}, 0x1, 0xc0000b6690)
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0001edf70, 0x3b9aca00, 0x0, 0x1, 0xc0000b6690)
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x1fc4ba0, 0xc000585b30}, 0xc0000548c0, 0x3b9aca00, 0x0, 0x1)
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x87
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
    /go/src/github.com/openshift/whereabouts/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170
created by github.com/k8snetworkplumbingwg/whereabouts/pkg/node-controller.(*Controller).Run in goroutine 1
    /go/src/github.com/openshift/whereabouts/pkg/node-controller/controller.go:227 +0x3ae    $ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.19.0-0.nightly-2025-03-30-191359   True        False         34m     Cluster version is 4.19.0-0.nightly-2025-03-30-191359

Expected results:

  whereabouts-controller pod is created and in Running state

Additional info:

links to

openshift/whereabouts-cni#356: OCPBUGS-54422: Ds merge 4/3

RHEA-2024:11038 OpenShift Container Platform 4.19.z bug fix update

Assignee:: Ben Pickard

Reporter:: Weibin Liang

Need Info From:: None

Contributors:: None

QA Contact:: Weibin Liang

Doc Contact:: None

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2025/03/31 2:07 PM

Updated:: 2025/07/14 1:27 PM

Resolved:: 2025/06/17 4:49 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates