One control plane node ( here xiyun-20-m-zxpf6-control-plane-1) cannot access the pod on one worker node(xiyun-20-m-zxpf6-compute-0).The profile used in the clusters: /upi-on-vsphere/versioned-installer-ovn-ipsec-static_network-bonding-active-backup-secureboot-fips-hw19-ci

4.14.0-0.nightly-2023-10-18-004928

Sometimes

1, create a pod on one worker node xiyun-20-m-zxpf6-compute-0
# oc apply -f pod.yaml 
pod/hello-pod created
# oc get pod hello-pod -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
hello-pod   1/1     Running   0          40s   10.131.0.65   xiyun-20-m-zxpf6-compute-0   <none>           <none>

2, Try to access this pod from master xiyun-20-m-zxpf6-control-plane-1, but failed to access it.
# oc debug node/xiyun-20-m-zxpf6-control-plane-1
Temporary namespace openshift-debug-h7dz8 is created for debugging node...
Starting pod/xiyun-20-m-zxpf6-control-plane-1-debug-qhbbr ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.221.88
If you don't see a command prompt, try pressing enter.
sh-4.4# curl 10.131.0.65:8080
^C
sh-4.4#


3. The access can be successful from other master nodes.
# oc debug node/xiyun-20-m-zxpf6-control-plane-0
Temporary namespace openshift-debug-kftgw is created for debugging node...
Starting pod/xiyun-20-m-zxpf6-control-plane-0-debug-g5f8f ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.221.89
If you don't see a command prompt, try pressing enter.
sh-4.4# curl 10.131.0.65:8080
Hello OpenShift!
sh-4.4# exit
exit
# oc debug node/xiyun-20-m-zxpf6-control-plane-2
Temporary namespace openshift-debug-t29j6 is created for debugging node...
Starting pod/xiyun-20-m-zxpf6-control-plane-2-debug-j8n9t ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.221.87
If you don't see a command prompt, try pressing enter.
sh-4.4# curl 10.131.0.65:8080
Hello OpenShift!
sh-4.4# exit
Exit

4. Created a pod on another work node xiyun-20-m-zxpf6-compute-1
 % oc get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-pod 1/1 Running 0 8s 10.128.2.20 xiyun-20-m-zxpf6-compute-1 <none> <none>
5. Try to access this pod from master node xiyun-20-m-zxpf6-control-plane-1, succeed.
# oc debug node/xiyun-20-m-zxpf6-control-plane-1
Starting pod/xiyun-20-m-zxpf6-control-plane-1-debug-gfwkz ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.221.88
If you don't see a command prompt, try pressing enter.
sh-4.4# curl 10.128.2.20:8080
Hello OpenShift!

One control plane node cannot access the pod on one worker node. Seen from the reproduce steps

The control plane node should able to access the pod on any worker node

Must gather logs are available at: https://drive.google.com/drive/folders/10NneU_ggcC992ySw1pvLj5qiVCEQk87R