The cluster was recently upgraded to 4.14 ODF OCP

 csi-addons-controller-manager cannot come up due to multus error

```
Apr 23 02:31:52 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:52.970704928Z" level=info msg="Running pod sandbox: openshift-storage/csi-addons-controller-manager-7df7cbf5f9-jz4sn/POD" id=88601335-8b5d-4a7b-bfd8-258d77f730ae name=/runtime.v1.RuntimeService/RunPodSandbox
Apr 23 02:31:52 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:52.970796730Z" level=warning msg="Allowed annotations are specified for workload [io.containers.trace-syscall]"
Apr 23 02:31:52 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:52.979085574Z" level=info msg="Got pod network &{Name:csi-addons-controller-manager-7df7cbf5f9-jz4sn Namespace:openshift-storage ID:5eb73ce2eaae232dc69133251524eba31a9efbc604fb02c88005cb6f5a43ff5f UID:34d36752-fed9-4a55-a966-a86279825f29 NetNS:/var/run/netns/19fcaed2-22f9-4532-8836-aa3fa6d09fa0 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}"
Apr 23 02:31:52 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:52.979119475Z" level=info msg="Adding pod openshift-storage_csi-addons-controller-manager-7df7cbf5f9-jz4sn to CNI network \"multus-cni-network\" (type=multus-shim)"
Apr 23 02:31:55 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:55.504394519Z" level=info msg="NetworkStart: stopping network for sandbox 5eb73ce2eaae232dc69133251524eba31a9efbc604fb02c88005cb6f5a43ff5f" id=88601335-8b5d-4a7b-bfd8-258d77f730ae name=/runtime.v1.RuntimeService/RunPodSandbox
Apr 23 02:31:55 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:55.504606123Z" level=info msg="Got pod network &{Name:csi-addons-controller-manager-7df7cbf5f9-jz4sn Namespace:openshift-storage ID:5eb73ce2eaae232dc69133251524eba31a9efbc604fb02c88005cb6f5a43ff5f UID:34d36752-fed9-4a55-a966-a86279825f29 NetNS:/var/run/netns/19fcaed2-22f9-4532-8836-aa3fa6d09fa0 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}"
Apr 23 02:31:55 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:55.504644724Z" level=error msg="error loading cached network config: network \"multus-cni-network\" not found in CNI cache"
Apr 23 02:31:55 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:55.504655524Z" level=warning msg="falling back to loading from existing plugins on disk"
Apr 23 02:31:55 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:55.504664824Z" level=info msg="Deleting pod openshift-storage_csi-addons-controller-manager-7df7cbf5f9-jz4sn from CNI network \"multus-cni-network\" (type=multus-shim)"
Apr 23 02:31:55 gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l crio[2469]: time="2024-04-23 02:31:55.522852541Z" level=info msg="runSandbox: cleaning up namespaces after failing to run sandbox 5eb73ce2eaae232dc69133251524eba31a9efbc604fb02c88005cb6f5a43ff5f" id=88601335-8b5d-4a7b-bfd8-258d77f730ae name=/runtime.v1.RuntimeService/RunPodSandbox

resulting in this:

oc get pods -l app.kubernetes.io/name=csi-addons -n openshift-storage -o wide -w
root@AP-22345344:~# oc get pods -l app.kubernetes.io/name=csi-addons -n openshift-storage -o wide -w
NAME                                             READY   STATUS              RESTARTS   AGE   IP       NODE                                                NOMINATED NODE   READINESS GATES
csi-addons-controller-manager-7df7cbf5f9-ftrs5   0/2     ContainerCreating   0          0s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-ftrs5   0/2     Terminating         0          1s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-ftrs5   0/2     Terminating         0          1s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-bw6x5   0/2     Pending             0          1s    <none>   <none>                                              <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-bw6x5   0/2     Pending             0          1s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-bw6x5   0/2     Pending             0          1s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-bw6x5   0/2     ContainerCreating   0          1s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-bw6x5   0/2     Terminating         0          1s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-bw6x5   0/2     Terminating         0          1s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-jrxmb   0/2     Pending             0          0s    <none>   <none>                                              <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-jrxmb   0/2     Pending             0          0s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-jrxmb   0/2     Pending             0          0s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>
csi-addons-controller-manager-7df7cbf5f9-jrxmb   0/2     ContainerCreating   0          0s    <none>   gfopenshiftacmpazwe-87vv9-odf03-westeurope3-74b4l   <none>           <none>

and as a result csi-addons is stuck installing

[vlours@supportshell-1 ~]$ oc get csv -n openshift-storage
NAME                                          DISPLAY                                    VERSION        REPLACES                                      PHASE
external-secrets-operator.v0.9.14             External Secrets Operator                  0.9.14         external-secrets-operator.v0.9.13             Succeeded
loki-operator.v5.9.0                          Loki Operator                              5.9.0          loki-operator.v5.8.5                          Succeeded
mcg-operator.v4.14.6-rhodf                    NooBaa Operator                            4.14.6-rhodf   mcg-operator.v4.14.5-rhodf                    Succeeded
ocs-operator.v4.14.6-rhodf                    OpenShift Container Storage                4.14.6-rhodf   ocs-operator.v4.14.5-rhodf                    Succeeded
odf-csi-addons-operator.v4.14.6-rhodf         CSI Addons                                 4.14.6-rhodf   odf-csi-addons-operator.v4.14.5-rhodf         Installing
odf-multicluster-orchestrator.v4.14.6-rhodf   ODF Multicluster Orchestrator              4.14.6-rhodf   odf-multicluster-orchestrator.v4.13.8-rhodf   Succeeded
odf-operator.v4.14.6-rhodf                    OpenShift Data Foundation                  4.14.6-rhodf   odf-operator.v4.14.5-rhodf                    Succeeded
odr-hub-operator.v4.14.6-rhodf                Openshift DR Hub Operator                  4.14.6-rhodf   odr-hub-operator.v4.13.8-rhodf                Succeeded
openshift-gitops-operator.v1.12.1             Red Hat OpenShift GitOps                   1.12.1         openshift-gitops-operator.v1.12.0             Succeeded
patch-operator.v0.1.9                         Patch Operator                             0.1.9          patch-operator.v0.1.8                         Succeeded
rhacs-operator.v4.4.0                         Advanced Cluster Security for Kubernetes   4.4.0          rhacs-operator.v4.3.5                         Succeeded
volsync-product.v0.7.4                        VolSync                                    0.7.4          volsync-product.v0.7.3                        Succeeded

    4.14

      We have engaged shift-network to look at the multus side and was advised to restart the multus pods, this did not resolve the error. 

    1.
    2.
    3.
    

      csi addons should complete installing
multus pods should not stop installation

      https://bugzilla.redhat.com/show_bug.cgi?id=2277045