Loading...

XML

Word

Printable

Type: Bug
Resolution: Done-Errata
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.13.0
Component/s: Networking / multus
Labels:
None

Regression:
No
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Target Version:

4.14.0

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Reproducer instructions:

You can set the whereabouts reconciler cronjob to run every minute with "*/1 * * * *"

oc patch clusterversion version --type json -p '[{"op":"add","path":"/spec/overrides","value":[{"kind":"Deployment","group":"apps","name":"network-operator","namespace":"openshift-network-operator","unmanaged":true}]}]'
oc scale -n openshift-network-operator deployment.apps/network-operator --replicas=0;

Then:

oc edit ds -n openshift-multus multus-additional-cni-plugins
(...)
 "reconciler_cron_expression": "*/1 * * * *"
(...)

Now, restart a single reconciler pod:

oc delete pod -n openshift-multus whereabouts-reconciler-n4m8j

The pod should now run the reconciler every minute.

Now, take a deployment, and scale it to 0 replicas:

[akaris@linux whereabouts-clusterbot]$ oc scale deployment netshoot-deployment --replicas 0
deployment.apps/netshoot-deployment scaled

Now, start the following loops:

 while true; do echo ===================== ; date ; oc get pods; oc get pods -o jsonpath='.items[].status'; oc get pods -o name | grep netshoot | while read p ; do oc exec $p -- ip a ls dev net1; done; done

 while true; do echo ==================;  date; oc get overlappingrangeipreservations -A -o yaml; oc get ippools -A -o yaml; sleep 5; done

oc logs -n openshift-multus whereabouts-reconciler-n4m8j --tail=0 -f

Scale to 3 replicas:

oc scale deployment netshoot-deployment --replicas 3

We can see now that the pods scale up ...:

akaris@linux whereabouts-clusterbot]$  while true; do echo ===================== ; date ; oc get pods; oc get pods -o jsonpath='.items[].status'; oc get pods -o name | grep netshoot | while read p ; do oc exec $p -- ip a ls dev net1; done; done
=====================
Wed Mar 15 01:58:14 PM CET 2023
No resources found in default namespace.
.items[].status=====================
Wed Mar 15 01:58:16 PM CET 2023
No resources found in default namespace.
.items[].status=====================
Wed Mar 15 01:58:18 PM CET 2023
No resources found in default namespace.
.items[].status=====================
Wed Mar 15 01:58:19 PM CET 2023
NAME                                   READY   STATUS              RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   0/1     ContainerCreating   0          1s
netshoot-deployment-6db7547555-smhc8   0/1     ContainerCreating   0          1s
netshoot-deployment-6db7547555-xhfvr   0/1     ContainerCreating   0          1s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:58:26 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          8s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          8s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          8s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:58:33 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          15s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          15s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          15s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:58:40 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          22s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          22s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          22s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:58:47 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          29s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          29s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          29s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:58:54 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          35s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          35s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          35s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:59:00 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          42s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          42s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          42s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:59:07 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          48s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          48s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          48s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:59:13 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          55s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          55s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          55s
.items[].status





3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever

... but that their allocations are erroneously removed:

[akaris@linux whereabouts-clusterbot]$ while true; do echo ==================;  date; oc get overlappingrangeipreservations -A -o yaml; oc get ippools -A -o yaml; sleep 5; done
==================
Wed Mar 15 01:58:12 PM CET 2023
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 14
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "60944"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations: {}
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:58:18 PM CET 2023
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 14
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "60944"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations: {}
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:58:24 PM CET 2023
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:20Z"
    generation: 1
    name: 192.168.123.20
    namespace: openshift-multus
    resourceVersion: "79302"
    uid: e052e83f-6e1c-4078-a6c0-a8febe80e49e
  spec:
    containerid: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
    podref: default/netshoot-deployment-6db7547555-xhfvr
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:21Z"
    generation: 1
    name: 192.168.123.21
    namespace: openshift-multus
    resourceVersion: "79320"
    uid: 8c573e81-f81e-46b0-bdbd-f47a01f7b5d9
  spec:
    containerid: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
    podref: default/netshoot-deployment-6db7547555-s89pm
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:22Z"
    generation: 1
    name: 192.168.123.22
    namespace: openshift-multus
    resourceVersion: "79342"
    uid: c7c2dfb9-18e7-4e2f-b2cf-49299c859774
  spec:
    containerid: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
    podref: default/netshoot-deployment-6db7547555-smhc8
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 17
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79341"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations:
      "20":
        id: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
        podref: default/netshoot-deployment-6db7547555-xhfvr
      "21":
        id: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
        podref: default/netshoot-deployment-6db7547555-s89pm
      "22":
        id: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
        podref: default/netshoot-deployment-6db7547555-smhc8
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:58:30 PM CET 2023
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:20Z"
    generation: 1
    name: 192.168.123.20
    namespace: openshift-multus
    resourceVersion: "79302"
    uid: e052e83f-6e1c-4078-a6c0-a8febe80e49e
  spec:
    containerid: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
    podref: default/netshoot-deployment-6db7547555-xhfvr
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:21Z"
    generation: 1
    name: 192.168.123.21
    namespace: openshift-multus
    resourceVersion: "79320"
    uid: 8c573e81-f81e-46b0-bdbd-f47a01f7b5d9
  spec:
    containerid: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
    podref: default/netshoot-deployment-6db7547555-s89pm
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:22Z"
    generation: 1
    name: 192.168.123.22
    namespace: openshift-multus
    resourceVersion: "79342"
    uid: c7c2dfb9-18e7-4e2f-b2cf-49299c859774
  spec:
    containerid: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
    podref: default/netshoot-deployment-6db7547555-smhc8
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 17
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79341"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations:
      "20":
        id: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
        podref: default/netshoot-deployment-6db7547555-xhfvr
      "21":
        id: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
        podref: default/netshoot-deployment-6db7547555-s89pm
      "22":
        id: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
        podref: default/netshoot-deployment-6db7547555-smhc8
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:58:36 PM CET 2023
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:20Z"
    generation: 1
    name: 192.168.123.20
    namespace: openshift-multus
    resourceVersion: "79302"
    uid: e052e83f-6e1c-4078-a6c0-a8febe80e49e
  spec:
    containerid: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
    podref: default/netshoot-deployment-6db7547555-xhfvr
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:21Z"
    generation: 1
    name: 192.168.123.21
    namespace: openshift-multus
    resourceVersion: "79320"
    uid: 8c573e81-f81e-46b0-bdbd-f47a01f7b5d9
  spec:
    containerid: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
    podref: default/netshoot-deployment-6db7547555-s89pm
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:22Z"
    generation: 1
    name: 192.168.123.22
    namespace: openshift-multus
    resourceVersion: "79342"
    uid: c7c2dfb9-18e7-4e2f-b2cf-49299c859774
  spec:
    containerid: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
    podref: default/netshoot-deployment-6db7547555-smhc8
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 17
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79341"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations:
      "20":
        id: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
        podref: default/netshoot-deployment-6db7547555-xhfvr
      "21":
        id: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
        podref: default/netshoot-deployment-6db7547555-s89pm
      "22":
        id: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
        podref: default/netshoot-deployment-6db7547555-smhc8
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:58:42 PM CET 2023
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:20Z"
    generation: 1
    name: 192.168.123.20
    namespace: openshift-multus
    resourceVersion: "79302"
    uid: e052e83f-6e1c-4078-a6c0-a8febe80e49e
  spec:
    containerid: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
    podref: default/netshoot-deployment-6db7547555-xhfvr
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:21Z"
    generation: 1
    name: 192.168.123.21
    namespace: openshift-multus
    resourceVersion: "79320"
    uid: 8c573e81-f81e-46b0-bdbd-f47a01f7b5d9
  spec:
    containerid: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
    podref: default/netshoot-deployment-6db7547555-s89pm
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:22Z"
    generation: 1
    name: 192.168.123.22
    namespace: openshift-multus
    resourceVersion: "79342"
    uid: c7c2dfb9-18e7-4e2f-b2cf-49299c859774
  spec:
    containerid: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
    podref: default/netshoot-deployment-6db7547555-smhc8
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 17
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79341"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations:
      "20":
        id: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
        podref: default/netshoot-deployment-6db7547555-xhfvr
      "21":
        id: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
        podref: default/netshoot-deployment-6db7547555-s89pm
      "22":
        id: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
        podref: default/netshoot-deployment-6db7547555-smhc8
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:58:48 PM CET 2023
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:20Z"
    generation: 1
    name: 192.168.123.20
    namespace: openshift-multus
    resourceVersion: "79302"
    uid: e052e83f-6e1c-4078-a6c0-a8febe80e49e
  spec:
    containerid: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
    podref: default/netshoot-deployment-6db7547555-xhfvr
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:21Z"
    generation: 1
    name: 192.168.123.21
    namespace: openshift-multus
    resourceVersion: "79320"
    uid: 8c573e81-f81e-46b0-bdbd-f47a01f7b5d9
  spec:
    containerid: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
    podref: default/netshoot-deployment-6db7547555-s89pm
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:22Z"
    generation: 1
    name: 192.168.123.22
    namespace: openshift-multus
    resourceVersion: "79342"
    uid: c7c2dfb9-18e7-4e2f-b2cf-49299c859774
  spec:
    containerid: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
    podref: default/netshoot-deployment-6db7547555-smhc8
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 17
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79341"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations:
      "20":
        id: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
        podref: default/netshoot-deployment-6db7547555-xhfvr
      "21":
        id: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
        podref: default/netshoot-deployment-6db7547555-s89pm
      "22":
        id: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
        podref: default/netshoot-deployment-6db7547555-smhc8
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:58:54 PM CET 2023
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:20Z"
    generation: 1
    name: 192.168.123.20
    namespace: openshift-multus
    resourceVersion: "79302"
    uid: e052e83f-6e1c-4078-a6c0-a8febe80e49e
  spec:
    containerid: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
    podref: default/netshoot-deployment-6db7547555-xhfvr
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:21Z"
    generation: 1
    name: 192.168.123.21
    namespace: openshift-multus
    resourceVersion: "79320"
    uid: 8c573e81-f81e-46b0-bdbd-f47a01f7b5d9
  spec:
    containerid: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
    podref: default/netshoot-deployment-6db7547555-s89pm
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:22Z"
    generation: 1
    name: 192.168.123.22
    namespace: openshift-multus
    resourceVersion: "79342"
    uid: c7c2dfb9-18e7-4e2f-b2cf-49299c859774
  spec:
    containerid: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
    podref: default/netshoot-deployment-6db7547555-smhc8
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 17
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79341"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations:
      "20":
        id: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
        podref: default/netshoot-deployment-6db7547555-xhfvr
      "21":
        id: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
        podref: default/netshoot-deployment-6db7547555-s89pm
      "22":
        id: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
        podref: default/netshoot-deployment-6db7547555-smhc8
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:59:00 PM CET 2023
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:20Z"
    generation: 1
    name: 192.168.123.20
    namespace: openshift-multus
    resourceVersion: "79302"
    uid: e052e83f-6e1c-4078-a6c0-a8febe80e49e
  spec:
    containerid: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
    podref: default/netshoot-deployment-6db7547555-xhfvr
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:21Z"
    generation: 1
    name: 192.168.123.21
    namespace: openshift-multus
    resourceVersion: "79320"
    uid: 8c573e81-f81e-46b0-bdbd-f47a01f7b5d9
  spec:
    containerid: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
    podref: default/netshoot-deployment-6db7547555-s89pm
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: OverlappingRangeIPReservation
  metadata:
    creationTimestamp: "2023-03-15T12:58:22Z"
    generation: 1
    name: 192.168.123.22
    namespace: openshift-multus
    resourceVersion: "79342"
    uid: c7c2dfb9-18e7-4e2f-b2cf-49299c859774
  spec:
    containerid: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
    podref: default/netshoot-deployment-6db7547555-smhc8
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 17
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79341"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations:
      "20":
        id: 87aead53324fc5d3346797a6c26ea438ccd0bbbf47bbb3e011d9b5522ab8926c
        podref: default/netshoot-deployment-6db7547555-xhfvr
      "21":
        id: 54e7773be81acb5a26a282960c96a055d74cd105e55d09528d0a3fd50c4d4e13
        podref: default/netshoot-deployment-6db7547555-s89pm
      "22":
        id: d075e6291dc82519f93205ce49eceed935dc3caa72196a4317684bb1276443d4
        podref: default/netshoot-deployment-6db7547555-smhc8
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:59:06 PM CET 2023
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 18
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79579"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations: {}
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""
==================
Wed Mar 15 01:59:12 PM CET 2023
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-03-15T11:30:14Z"
    generation: 18
    name: 192.168.123.0-24
    namespace: openshift-multus
    resourceVersion: "79579"
    uid: c8a4f758-17fc-4177-8f0a-dd2894f9271f
  spec:
    allocations: {}
    range: 192.168.123.0/24
kind: List
metadata:
  resourceVersion: ""

[akaris@linux whereabouts-clusterbot]$ oc logs -n openshift-multus whereabouts-reconciler-n4m8j --tail=0 -f

2023-03-15T12:59:00Z [verbose] starting reconciler run
2023-03-15T12:59:00Z [debug] NewReconcileLooper - inferred connection data
2023-03-15T12:59:00Z [debug] listing IP pools
2023-03-15T12:59:02Z [debug] the IP reservation: IP: 192.168.123.20 is reserved for pod: default/netshoot-deployment-6db7547555-xhfvr
2023-03-15T12:59:02Z [debug] pod reference default/netshoot-deployment-6db7547555-xhfvr matches allocation; Allocation IP: 192.168.123.20; PodIPs: map[]
2023-03-15T12:59:02Z [debug] pod ref default/netshoot-deployment-6db7547555-xhfvr is not listed in the live pods list
2023-03-15T12:59:02Z [debug] the IP reservation: IP: 192.168.123.21 is reserved for pod: default/netshoot-deployment-6db7547555-s89pm
2023-03-15T12:59:02Z [debug] pod reference default/netshoot-deployment-6db7547555-s89pm matches allocation; Allocation IP: 192.168.123.21; PodIPs: map[]
2023-03-15T12:59:02Z [debug] pod ref default/netshoot-deployment-6db7547555-s89pm is not listed in the live pods list
2023-03-15T12:59:02Z [debug] the IP reservation: IP: 192.168.123.22 is reserved for pod: default/netshoot-deployment-6db7547555-smhc8
2023-03-15T12:59:02Z [debug] pod reference default/netshoot-deployment-6db7547555-smhc8 matches allocation; Allocation IP: 192.168.123.22; PodIPs: map[]
2023-03-15T12:59:02Z [debug] pod ref default/netshoot-deployment-6db7547555-smhc8 is not listed in the live pods list
2023-03-15T12:59:02Z [debug] pod reference default/netshoot-deployment-6db7547555-xhfvr matches allocation; Allocation IP: 192.168.123.20; PodIPs: map[]
2023-03-15T12:59:02Z [debug] pod ref default/netshoot-deployment-6db7547555-xhfvr is not listed in the live pods list
2023-03-15T12:59:02Z [debug] pod reference default/netshoot-deployment-6db7547555-s89pm matches allocation; Allocation IP: 192.168.123.21; PodIPs: map[]
2023-03-15T12:59:02Z [debug] pod ref default/netshoot-deployment-6db7547555-s89pm is not listed in the live pods list
2023-03-15T12:59:02Z [debug] pod reference default/netshoot-deployment-6db7547555-smhc8 matches allocation; Allocation IP: 192.168.123.22; PodIPs: map[]
2023-03-15T12:59:02Z [debug] pod ref default/netshoot-deployment-6db7547555-smhc8 is not listed in the live pods list
2023-03-15T12:59:02Z [debug] Going to update the reserve list to: []
2023-03-15T12:59:02Z [debug] successfully cleanup IPs: [192.168.123.22]
2023-03-15T12:59:02Z [verbose] removed stale overlappingIP allocation [192.168.123.20]
2023-03-15T12:59:02Z [verbose] removed stale overlappingIP allocation [192.168.123.21]
2023-03-15T12:59:02Z [verbose] removed stale overlappingIP allocation [192.168.123.22]
2023-03-15T12:59:02Z [verbose] reconciler success

Now, scale to 6 replicas:

oc scale deployment netshoot-deployment --replicas 6

And we can see now that due to the missing allocations, the next batch of pods is assigned the same IPs and we end up with duplicate IP assignments:

3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:59:20 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          61s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          61s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          61s
.items[].status3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:59:27 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-ffvt4   1/1     Running   0          4s
netshoot-deployment-6db7547555-mkq28   1/1     Running   0          4s
netshoot-deployment-6db7547555-pgtcg   1/1     Running   0          4s
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          68s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          68s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          68s
.items[].status3: net1@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 86:75:87:e8:60:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::8475:87ff:fee8:6030/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether ea:c7:8f:81:11:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::e8c7:8fff:fe81:11c3/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c6:49:fa:0f:9a:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::c449:faff:fe0f:9ae4/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:59:38 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-ffvt4   1/1     Running   0          15s
netshoot-deployment-6db7547555-mkq28   1/1     Running   0          15s
netshoot-deployment-6db7547555-pgtcg   1/1     Running   0          15s
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          79s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          79s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          79s
.items[].status3: net1@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 86:75:87:e8:60:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::8475:87ff:fee8:6030/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether ea:c7:8f:81:11:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::e8c7:8fff:fe81:11c3/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c6:49:fa:0f:9a:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::c449:faff:fe0f:9ae4/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 01:59:49 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-ffvt4   1/1     Running   0          26s
netshoot-deployment-6db7547555-mkq28   1/1     Running   0          26s
netshoot-deployment-6db7547555-pgtcg   1/1     Running   0          26s
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          90s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          90s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          90s
.items[].status3: net1@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 86:75:87:e8:60:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::8475:87ff:fee8:6030/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether ea:c7:8f:81:11:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::e8c7:8fff:fe81:11c3/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c6:49:fa:0f:9a:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::c449:faff:fe0f:9ae4/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:8c:9a:e5:79:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.21/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::248c:9aff:fee5:79e6/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:58:1f:9d:50:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5058:1fff:fe9d:50ca/64 scope link 
       valid_lft forever preferred_lft forever
3: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 22:28:95:11:2e:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.20/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::2028:95ff:fe11:2eaf/64 scope link 
       valid_lft forever preferred_lft forever
=====================
Wed Mar 15 02:00:01 PM CET 2023
NAME                                   READY   STATUS    RESTARTS   AGE
netshoot-deployment-6db7547555-ffvt4   1/1     Running   0          38s
netshoot-deployment-6db7547555-mkq28   1/1     Running   0          38s
netshoot-deployment-6db7547555-pgtcg   1/1     Running   0          38s
netshoot-deployment-6db7547555-s89pm   1/1     Running   0          102s
netshoot-deployment-6db7547555-smhc8   1/1     Running   0          102s
netshoot-deployment-6db7547555-xhfvr   1/1     Running   0          102s
.items[].status3: net1@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 86:75:87:e8:60:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.123.22/24 brd 192.168.123.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::8475:87ff:fee8:6030/64 scope link 
       valid_lft forever preferred_lft forever
^C
[akaris@linux whereabouts-clusterbot]$

Root cause

This happens because the pods annotation is:

  annotations:
    k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"ip_address":"10.131.0.26/23","gateway_ip":"10.131.0.1"}}'
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.131.0.26"
          ],
          "mac": "0a:58:0a:83:00:1a",
          "default": true,
          "dns": {}
      },{
          "name": "default/br1-net",
          "interface": "net1",
          "ips": [
              "192.168.123.22"
          ],
          "mac": "86:75:87:e8:60:30",
          "dns": {}
      }]

But this code here:

   92 func networkStatusFromPod(pod v1.Pod) string {                                                                          
   93     networkStatusAnnotationValue, isStatusAnnotationPresent := pod.Annotations[MultusNetworkStatusAnnotation]           
   94     if !isStatusAnnotationPresent || len(networkStatusAnnotationValue) == 0 {                                           
   95         return "[]"                                                                                                     
   96     }                                                                                                                   
   97     return networkStatusAnnotationValue                                                                                 
   98 }

Is looking for:

MultusNetworkStatusAnnotation = "k8s.v1.cni.cncf.io/networks-status"

Note network-status vs networks-status

is cloned by

OCPBUGS-11427 [release-4.13] whereabouts reads wrong annotation "k8s.v1.cni.cncf.io/networks-status", should be "k8s.v1.cni.cncf.io/network-status"

Closed

is depended on by

OCPBUGS-11427 [release-4.13] whereabouts reads wrong annotation "k8s.v1.cni.cncf.io/networks-status", should be "k8s.v1.cni.cncf.io/network-status"

Closed

links to

Downstream whereabouts fix (upstream -> downstream merge)

RHEA-2023:5006 rpm

upstream PR

Assignee:: Andreas Karis

Reporter:: Andreas Karis

QA Contact:: Weibin Liang

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2023/03/15 1:03 PM

Updated:: 2023/10/31 1:33 PM

Resolved:: 2023/10/31 12:58 PM

Details

Description

Reproducer instructions:

Root cause

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates