Network checkup pod is not deleted even the checkup is deleted

CNV-v4.15.0.rhel9-2585

Always

1. Access the console
2. Click Virtualization -> Checkups 
3. Run network checkup - checkup failed
4. Delete the checkup

$ oc get pod
NAME                                                         READY   STATUS    RESTARTS   AGE
kubevirt-vm-latency-checkup-peach-anglerfish-31-9472-9wbgv   0/1     Error     0          18m

The checkup pod should be deleted if the checkup is deleted

[cloud-user@ocp-psi-executor cnv-tests]$ oc get pod
NAME                                                         READY   STATUS    RESTARTS   AGE
kubevirt-storage-checkup-red-wombat-82-4102-2z8ms            0/1     Error     0          4m13s
kubevirt-vm-latency-checkup-peach-anglerfish-31-9472-9wbgv   0/1     Error     0          18m
virt-launcher-vmi-under-test-wtk6p-892wp                     1/1     Running   0          5m47s
[cloud-user@ocp-psi-executor cnv-tests]$ oc describe pod kubevirt-vm-latency-checkup-peach-anglerfish-31-9472-9wbgv
Name:             kubevirt-vm-latency-checkup-peach-anglerfish-31-9472-9wbgv
Namespace:        default
Priority:         0
Service Account:  vm-latency-checkup-sa
Node:             c01-yadu415-rcpj2-worker-0-2ldhq/192.168.0.246
Start Time:       Wed, 21 Feb 2024 04:46:25 -0500
Labels:           batch.kubernetes.io/controller-uid=48ff10d3-d7df-4231-84cc-6cdcbe7c87e6
                  batch.kubernetes.io/job-name=kubevirt-vm-latency-checkup-peach-anglerfish-31-9472
                  controller-uid=48ff10d3-d7df-4231-84cc-6cdcbe7c87e6
                  job-name=kubevirt-vm-latency-checkup-peach-anglerfish-31-9472
Annotations:      k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["10.131.0.87/23"],"mac_address":"0a:58:0a:83:00:57","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "10.131.0.87"
                        ],
                        "mac": "0a:58:0a:83:00:57",
                        "default": true,
                        "dns": {}
                    }]
Status:           Failed
IP:               10.131.0.87
IPs:
  IP:           10.131.0.87
Controlled By:  Job/kubevirt-vm-latency-checkup-peach-anglerfish-31-9472
Containers:
  vm-latency-checkup-sa:
    Container ID:   cri-o://fbfac1074355e87ca7bd7f5d42e509fc8e2d99a061e34ad0b9d5dd6b1f5e3f31
    Image:          registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.13.0
    Image ID:       registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9@sha256:59fd965975bccf641a0fa7aa01374d45057bd03cda7528248a84b60b1205bf57
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 21 Feb 2024 04:46:35 -0500
      Finished:     Wed, 21 Feb 2024 04:51:45 -0500
    Ready:          False
    Restart Count:  0
    Environment:
      CONFIGMAP_NAMESPACE:  default
      CONFIGMAP_NAME:       kubevirt-vm-latency-checkup-peach-anglerfish-31
      POD_UID:               (v1:metadata.uid)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-djlcn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-djlcn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       22m   default-scheduler  Successfully assigned default/kubevirt-vm-latency-checkup-peach-anglerfish-31-9472-9wbgv to c01-yadu415-rcpj2-worker-0-2ldhq
  Normal  AddedInterface  22m   multus             Add eth0 [10.131.0.87/23] from ovn-kubernetes
  Normal  Pulling         22m   kubelet            Pulling image "registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.13.0"
  Normal  Pulled          22m   kubelet            Successfully pulled image "registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.13.0" in 8.838s (8.838s including waiting)
  Normal  Created         22m   kubelet            Created container vm-latency-checkup-sa
  Normal  Started         22m   kubelet            Started container vm-latency-checkup-sa
[cloud-user@ocp-psi-executor cnv-tests]$ oc logs kubevirt-vm-latency-checkup-peach-anglerfish-31-9472-9wbgv
2024/02/21 09:46:35 starting VMI default/latency-check-source-kg5c5..
W0221 09:46:35.805341       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
2024/02/21 09:46:35 starting VMI default/latency-check-target-f9bf9..
W0221 09:46:35.833026       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
2024/02/21 09:46:35 waiting for VMI default/latency-check-target-f9bf9 IP address to appear on status..
W0221 09:46:35.855224       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:46:40.863028       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:46:45.862053       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:46:50.866160       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:46:55.861727       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:00.862927       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:05.860802       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:10.862650       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:15.861937       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:20.868415       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:25.862944       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:30.867035       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:35.872071       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:40.902153       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:45.870883       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:50.862254       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:47:55.864492       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:00.864097       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:05.862867       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:10.871211       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:15.864192       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:20.878911       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:25.862676       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:30.863189       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:35.861452       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:40.861587       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:45.862745       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:50.874866       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:48:55.861347       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:00.869631       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:05.861466       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:10.863519       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:15.863628       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:20.863855       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:25.862183       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:30.862865       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:35.872497       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:40.868419       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:45.861576       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:50.917601       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:49:55.863546       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:00.868295       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:05.862303       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:10.862658       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:15.867193       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:20.862606       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:25.861786       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:30.862159       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:35.863865       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:40.878856       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:45.874974       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:50.866496       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:50:55.863487       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:00.861847       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:05.863105       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:10.879447       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:15.861501       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:20.866892       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:25.862194       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:30.863583       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
2024/02/21 09:51:35 setup failed, cleanup VMI 'default/latency-check-target-f9bf9'
2024/02/21 09:51:35 deleting VMI default/latency-check-target-f9bf9..
W0221 09:51:35.763584       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
2024/02/21 09:51:35 waiting for VMI default/latency-check-target-f9bf9 to dispose..
W0221 09:51:35.768577       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:40.793149       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
2024/02/21 09:51:40 setup failed, cleanup VMI 'default/latency-check-source-kg5c5'
2024/02/21 09:51:40 deleting VMI default/latency-check-source-kg5c5..
W0221 09:51:40.816913       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
2024/02/21 09:51:40 waiting for VMI default/latency-check-source-kg5c5 to dispose..
W0221 09:51:40.820897       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
W0221 09:51:45.825705       1 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
2024/02/21 09:51:45 reporting status:
{}
2024/02/21 09:51:45 Kubevirt VM latency checkup failed: setup: failed to wait for VMI 'default/latency-check-target-f9bf9' IP address to appear on status: timed out waiting for the condition