Description of problem: rdma-device field is not updated in pod annotations with rdma plugin for sriovNetwork when sriov-device-plugin is restarted 

Version-Release number of selected component (if applicable): 4.18.ec3

How reproducible: 100%

Steps to Reproduce: 
1. Create Sriov Policy for 2VFs and Create Sriov Network
2. Create a pod with 2 VFs
3. In pod network annotations of VF, pci-address and rdma-device are updated.
4. Restart sriov-device-plugin pod that shares same worker node as test pod.
5. Delete the test pod and recreate same pod to re-use the same VFs.
6. Now the pod annotations does not have "rdma-device" field updated for the VFs.

Actual results:
"rdma-device" field not updated in pod network annotations 

Expected results:
"rdma-device" field to be populated in pod network annotations

Additional info: