Pod static IP is reachable after pod deletion for approx 1 minute    

4.16.0-rc.2
sriov-network-operator.v4.16.0-202405161711    

100%

    1. Create following sriovnetworknodepolicy and sriovnetwork:
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  annotations:
    operator.sriovnetwork.openshift.io/last-network-namespace: ztp-testns
  creationTimestamp: "2024-05-21T09:05:42Z"
  finalizers:
  - netattdef.finalizers.sriovnetwork.openshift.io
  generation: 1
  name: sriov-net-1
  namespace: openshift-sriov-network-operator
  resourceVersion: "444631"
  uid: 8e0006d0-836c-477a-bc3b-a286af97ebe4
spec:
  capabilities: '{"ips": true}'
  ipam: '{"type": "static","addresses": [{"address": "2620:52:0:199::/64","gateway":
    "2620:52:0:199::fe"}]}'
  logLevel: info
  networkNamespace: ztp-testns
  resourceName: pci_sriov_net_f1
  vlan: 107

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  creationTimestamp: "2024-05-20T11:02:06Z"
  generation: 1
  name: pci-sriov-net-f1
  namespace: openshift-sriov-network-operator
  resourceVersion: "37810"
  uid: 215fd74e-8a9b-4d23-a901-4761469f6ee0
spec:
  deviceType: netdevice
  isRdma: false
  nicSelector:
    pfNames:
    - ens1f2#0-1
  nodeSelector:
    node-role.kubernetes.io/master: ""
  numVfs: 8
  resourceName: pci_sriov_net_f1

2. Create the following pod:

apiVersion: v1
kind: Pod
metadata:
  name: sriov-1vf
  namespace: "ztp-testns"
  labels:
     app: sriov-1vf
  annotations:
    k8s.v1.cni.cncf.io/networks: |-
     [
      {
        "name": "sriov-net-1",
        "namespace": "ztp-testns",
        "ips": ["2620:52:0:199::f/64"]
      }
     ]
spec:
  containers:
  - name: sriov-1vfs
    image: registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/openshift/network-tools-rhel8:v4.11.0
    imagePullPolicy: IfNotPresent
    readinessProbe:
      exec:
        command:
        - sh
        - -c
        - ping -w1 -c1 2620:52:0:199::f
      initialDelaySeconds: 10
      periodSeconds: 5
      successThreshold: 1
      failureThreshold: 1
      timeoutSeconds: 15
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 300000; done;" ]
    securityContext:
      runAsGroup: 9999
      runAsUser: 9999
      capabilities:
        add: ["NET_ADMIN"]
    resources:
      limits:
        openshift.io/pci_sriov_net_f1: "2"
      requests:
        openshift.io/pci_sriov_net_f1: "2"     

3. Wait for the pod to become Ready

4. Delete the pod

5. Wait for the pod to be deleted

6. Verify network connectivity to the deleted pod static address
    

[kni@registry.kni-qe-0 ~]$ oc apply -f /home/kni/test_ztp_du_manifests/test_ztp_du_sriov_1vf_with_traffic_pod.yaml;date; oc -n ztp-testns wait --timeout=300s --for=condition=Ready=true pods --all;date; oc -n ztp-testns delete pods sriov-1vf;date; oc -n ztp-testns get pods;date; ping 2620:52:0:199::f | while read result; do echo "[$(date)] $result";done
pod/sriov-1vf created
Tue May 21 10:21:57 UTC 2024
pod/sriov-1vf condition met
Tue May 21 10:22:13 UTC 2024
pod "sriov-1vf" deleted
Tue May 21 10:22:45 UTC 2024
No resources found in ztp-testns namespace.
Tue May 21 10:22:45 UTC 2024
[Tue May 21 10:22:45 UTC 2024] PING 2620:52:0:199::f(2620:52:0:199::f) 56 data bytes
[Tue May 21 10:22:45 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=1 ttl=64 time=0.159 ms
[Tue May 21 10:22:46 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=2 ttl=64 time=0.050 ms
[Tue May 21 10:22:47 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=3 ttl=64 time=0.079 ms
[Tue May 21 10:22:48 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=4 ttl=64 time=0.061 ms
[Tue May 21 10:22:49 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=5 ttl=64 time=0.061 ms
[Tue May 21 10:22:50 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=6 ttl=64 time=0.082 ms
[Tue May 21 10:22:51 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=7 ttl=64 time=0.081 ms
[Tue May 21 10:22:52 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=8 ttl=64 time=0.076 ms
[Tue May 21 10:22:53 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=9 ttl=64 time=0.074 ms
[Tue May 21 10:22:54 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=10 ttl=64 time=0.072 ms
[Tue May 21 10:22:55 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=11 ttl=64 time=0.077 ms
[Tue May 21 10:22:56 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=12 ttl=64 time=0.075 ms
[Tue May 21 10:22:57 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=13 ttl=64 time=0.076 ms
[Tue May 21 10:22:58 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=14 ttl=64 time=0.079 ms
[Tue May 21 10:22:59 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=15 ttl=64 time=0.096 ms
[Tue May 21 10:23:00 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=16 ttl=64 time=0.094 ms
[Tue May 21 10:23:01 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=17 ttl=64 time=0.078 ms
[Tue May 21 10:23:02 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=18 ttl=64 time=0.073 ms
[Tue May 21 10:23:03 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=19 ttl=64 time=0.087 ms
[Tue May 21 10:23:04 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=20 ttl=64 time=0.075 ms
[Tue May 21 10:23:05 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=21 ttl=64 time=0.047 ms
[Tue May 21 10:23:06 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=22 ttl=64 time=0.078 ms
[Tue May 21 10:23:07 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=23 ttl=64 time=0.074 ms
[Tue May 21 10:23:08 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=24 ttl=64 time=0.076 ms
[Tue May 21 10:23:09 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=25 ttl=64 time=0.073 ms
[Tue May 21 10:23:10 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=26 ttl=64 time=0.075 ms
[Tue May 21 10:23:11 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=27 ttl=64 time=0.075 ms
[Tue May 21 10:23:13 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=28 ttl=64 time=0.074 ms
[Tue May 21 10:23:14 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=29 ttl=64 time=0.074 ms
[Tue May 21 10:23:15 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=30 ttl=64 time=0.095 ms
[Tue May 21 10:23:16 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=31 ttl=64 time=0.078 ms
[Tue May 21 10:23:17 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=32 ttl=64 time=0.075 ms
[Tue May 21 10:23:18 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=33 ttl=64 time=0.076 ms
[Tue May 21 10:23:19 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=34 ttl=64 time=0.074 ms
[Tue May 21 10:23:20 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=35 ttl=64 time=0.076 ms
[Tue May 21 10:23:21 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=36 ttl=64 time=0.065 ms
[Tue May 21 10:23:22 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=37 ttl=64 time=0.082 ms
[Tue May 21 10:23:23 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=38 ttl=64 time=0.072 ms
[Tue May 21 10:23:24 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=39 ttl=64 time=0.072 ms
[Tue May 21 10:23:25 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=40 ttl=64 time=0.070 ms
[Tue May 21 10:23:26 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=41 ttl=64 time=0.072 ms
[Tue May 21 10:23:27 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=42 ttl=64 time=0.074 ms
[Tue May 21 10:23:38 UTC 2024] From 2620:52:0:199::1 icmp_seq=50 Destination unreachable: Address unreachable
[Tue May 21 10:23:38 UTC 2024] From 2620:52:0:199::1 icmp_seq=51 Destination unreachable: Address unreachable
^C
[kni@registry.kni-qe-0 ~]$ oc apply -f /home/kni/test_ztp_du_manifests/test_ztp_du_sriov_1vf_with_traffic_pod.yaml;date; oc -n ztp-testns wait --timeout=300s --for=condition=Ready=true pods --all;date; oc -n ztp-testns delete pods sriov-1vf;date; oc -n ztp-testns get pods;date; ping 2620:52:0:199::f | while read result; do echo "[$(date)] $result";done
pod/sriov-1vf created
Tue May 21 10:23:59 UTC 2024
pod/sriov-1vf condition met
Tue May 21 10:24:15 UTC 2024
pod "sriov-1vf" deleted
Tue May 21 10:24:46 UTC 2024
No resources found in ztp-testns namespace.
Tue May 21 10:24:46 UTC 2024
[Tue May 21 10:24:46 UTC 2024] PING 2620:52:0:199::f(2620:52:0:199::f) 56 data bytes
[Tue May 21 10:24:46 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=1 ttl=64 time=0.159 ms
[Tue May 21 10:24:47 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=2 ttl=64 time=0.083 ms
[Tue May 21 10:24:49 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=3 ttl=64 time=0.060 ms
[Tue May 21 10:24:50 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=4 ttl=64 time=0.072 ms
[Tue May 21 10:24:51 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=5 ttl=64 time=0.072 ms
[Tue May 21 10:24:52 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=6 ttl=64 time=0.075 ms
[Tue May 21 10:24:53 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=7 ttl=64 time=0.077 ms
[Tue May 21 10:24:54 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=8 ttl=64 time=0.076 ms
[Tue May 21 10:24:55 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=9 ttl=64 time=0.074 ms
[Tue May 21 10:24:56 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=10 ttl=64 time=0.069 ms
[Tue May 21 10:24:57 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=11 ttl=64 time=0.071 ms
[Tue May 21 10:24:58 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=12 ttl=64 time=0.076 ms
[Tue May 21 10:24:59 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=13 ttl=64 time=0.074 ms
[Tue May 21 10:25:00 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=14 ttl=64 time=0.094 ms
[Tue May 21 10:25:01 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=15 ttl=64 time=0.073 ms
[Tue May 21 10:25:02 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=16 ttl=64 time=0.077 ms
[Tue May 21 10:25:03 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=17 ttl=64 time=0.083 ms
[Tue May 21 10:25:04 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=18 ttl=64 time=0.075 ms
[Tue May 21 10:25:05 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=19 ttl=64 time=0.070 ms
[Tue May 21 10:25:06 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=20 ttl=64 time=0.073 ms
[Tue May 21 10:25:07 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=21 ttl=64 time=0.075 ms
[Tue May 21 10:25:08 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=22 ttl=64 time=0.074 ms
[Tue May 21 10:25:09 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=23 ttl=64 time=0.077 ms
[Tue May 21 10:25:10 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=24 ttl=64 time=0.076 ms
[Tue May 21 10:25:11 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=25 ttl=64 time=0.072 ms
[Tue May 21 10:25:12 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=26 ttl=64 time=0.072 ms
[Tue May 21 10:25:13 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=27 ttl=64 time=0.073 ms
[Tue May 21 10:25:14 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=28 ttl=64 time=0.071 ms
[Tue May 21 10:25:15 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=29 ttl=64 time=0.087 ms
[Tue May 21 10:25:16 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=30 ttl=64 time=0.078 ms
[Tue May 21 10:25:17 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=31 ttl=64 time=0.078 ms
[Tue May 21 10:25:18 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=32 ttl=64 time=0.079 ms
[Tue May 21 10:25:19 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=33 ttl=64 time=0.071 ms
[Tue May 21 10:25:20 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=34 ttl=64 time=0.075 ms
[Tue May 21 10:25:21 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=35 ttl=64 time=0.096 ms
[Tue May 21 10:25:22 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=36 ttl=64 time=0.074 ms
[Tue May 21 10:25:23 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=37 ttl=64 time=0.076 ms
[Tue May 21 10:25:24 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=38 ttl=64 time=0.076 ms
[Tue May 21 10:25:25 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=39 ttl=64 time=0.072 ms
[Tue May 21 10:25:26 UTC 2024] 64 bytes from 2620:52:0:199::f: icmp_seq=40 ttl=64 time=0.075 ms
[Tue May 21 10:25:39 UTC 2024] From 2620:52:0:199::1 icmp_seq=49 Destination unreachable: Address unreachable
[Tue May 21 10:25:39 UTC 2024] From 2620:52:0:199::1 icmp_seq=50 Destination unreachable: Address unreachable   

The pod's static IP address is unreachable as the pod was deleted.    

This behavior seems to have been recently introduced with 4.16.0-rc.2. It has not been observed with earlier releases.

Attaching must-gather.