Description of problem:

Multus can error out with:

    pod was already deleted: SetPodNetworkStatusAnnotation: failed to query the pod revision-pruner-10-master-2 in out of cluster comm: pod "revision-pruner-10-master-2" not found

see: this test job for the example

I believe this occurs during a CNI ADD, the pod is deleted before Multus completes its CNI ADD operations, and then fails at this point. Potentially we can handle this case more cleanly.

Additionally, pods can also cause an undue amount of noise when connectivity to the API server isn't available. Such as this CI result and this other CI result, where we see errors like: "dial tcp 192.168.111.5:6443: connect: connection refused" – this will take some more careful consideration.

Version-Release number of selected component (if applicable):

    4.19 (and likely earlier...)

How reproducible:

See: sippy search

Challenging to reproduce.

Steps to Reproduce:

(not available at this time)
    

Actual results:

CI shows errors.    

Expected results:

Multus finishes cleanly in the case that a pod is already deleted. Or the connectivity to the API server is lost.

Additional info: