Description of problem: The frame.len difference issue when the packet passes through the VXLAN tunnel

Version-Release number of selected component (if applicable): 4.14.18

How reproducible: Intermittent

Additional info:
The customer has upgraded the cluster to 4.14.18 (OpenShiftSDN) on vSphere and observed the intermittent timeout while connecting the external database via EIP. We have observed the frame.len difference for the problematic sequence when it passes through the VXLAN. If the Pod is running on the EIP node, no issues are observed.
 
$ TZ=Asia/Kolkata tshark -T fields -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e tcp.time_delta -e frame.len -e _ws.col.Info -tad -Y tcp.seq==793219 -r 0360-vxlan_sys_4789_scposw1075.ocpprd01.prd.isaq.app_23_04_2024-20_59_33-UTC.pcap Apr 24, 2024 02:30:54.254909000 IST 10.131.1.163 53884 172.16.97.16 60728 0.000743000 243 OPNQRY | SQLDTA Apr 24, 2024 02:30:54.465854000 IST 10.131.1.163 53884 172.16.97.16 60728 0.210945000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536042079 TSecr=1559639159 Apr 24, 2024 02:30:54.672902000 IST 10.131.1.163 53884 172.16.97.16 60728 0.207048000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536042286 TSecr=1559639159 Apr 24, 2024 02:30:55.080903000 IST 10.131.1.163 53884 172.16.97.16 60728 0.408001000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536042694 TSecr=1559639159 Apr 24, 2024 02:30:55.945868000 IST 10.131.1.163 53884 172.16.97.16 60728 0.864965000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536043559 TSecr=1559639159 Apr 24, 2024 02:30:57.608865000 IST 10.131.1.163 53884 172.16.97.16 60728 1.662997000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536045222 TSecr=1559639159 Apr 24, 2024 02:31:00.872841000 IST 10.131.1.163 53884 172.16.97.16 60728 3.263976000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536048486 TSecr=1559639159 Apr 24, 2024 02:31:07.720941000 IST 10.131.1.163 53884 172.16.97.16 60728 6.848100000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536055334 TSecr=1559639159 Apr 24, 2024 02:31:21.034203000 IST 10.131.1.163 53884 172.16.97.16 60728 13.313262000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536068647 TSecr=1559639159 Apr 24, 2024 02:31:47.144884000 IST 10.131.1.163 53884 172.16.97.16 60728 5.710491000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536094758 TSecr=1559639159 Apr 24, 2024 02:32:41.929022000 IST 10.131.1.163 53884 172.16.97.16 60728 13.902641000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536149542 TSecr=1559639159 Apr 24, 2024 02:34:28.424969000 IST 10.131.1.163 53884 172.16.97.16 60728 4.174336000 243 [TCP Retransmission] 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=177 TSval=536256038 TSecr=1559639159
 
$ TZ=Asia/Kolkata tshark -T fields -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e tcp.time_delta -e frame.len -e _ws.col.Info -tad -Y tcp.seq==793219 -r 0400-vxlan_sys_4789_scposi1067.ocpprd01.prd.isaq.app_23_04_2024-20_59_48-UTC.pcap Apr 24, 2024 02:30:54.255259000 IST 10.131.1.163 53884 172.16.97.16 60728 0.001404000 66 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536041868 TSecr=1559639159 Apr 24, 2024 02:30:54.465995000 IST 10.131.1.163 53884 172.16.97.16 60728 0.210736000 66 TCP Dup ACK 42457#1 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536042079 TSecr=1559639159 Apr 24, 2024 02:30:54.673010000 IST 10.131.1.163 53884 172.16.97.16 60728 0.207015000 66 TCP Dup ACK 42457#2 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536042286 TSecr=1559639159 Apr 24, 2024 02:30:55.081030000 IST 10.131.1.163 53884 172.16.97.16 60728 0.408020000 66 TCP Dup ACK 42457#3 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536042694 TSecr=1559639159 Apr 24, 2024 02:30:55.945990000 IST 10.131.1.163 53884 172.16.97.16 60728 0.864960000 66 TCP Dup ACK 42457#4 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536043559 TSecr=1559639159 Apr 24, 2024 02:30:57.609040000 IST 10.131.1.163 53884 172.16.97.16 60728 1.663050000 66 TCP Dup ACK 42457#5 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536045222 TSecr=1559639159 Apr 24, 2024 02:31:00.872937000 IST 10.131.1.163 53884 172.16.97.16 60728 3.263897000 66 TCP Dup ACK 42457#6 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536048486 TSecr=1559639159 Apr 24, 2024 02:31:07.721068000 IST 10.131.1.163 53884 172.16.97.16 60728 6.848131000 66 TCP Dup ACK 42457#7 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536055334 TSecr=1559639159 Apr 24, 2024 02:31:21.034418000 IST 10.131.1.163 53884 172.16.97.16 60728 13.313350000 66 TCP Dup ACK 42457#8 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536068647 TSecr=1559639159 Apr 24, 2024 02:31:47.146214000 IST 10.131.1.163 53884 172.16.97.16 60728 5.710491000 66 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536094758 TSecr=1559639159 Apr 24, 2024 02:32:41.930805000 IST 10.131.1.163 53884 172.16.97.16 60728 13.902750000 66 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536149542 TSecr=1559639159 Apr 24, 2024 02:34:28.425988000 IST 10.131.1.163 53884 172.16.97.16 60728 4.173372000 66 53884 → 60728 [PSH, ACK] Seq=793219 Ack=1044652 Win=1277056 Len=0 TSval=536256038 TSecr=1559639159

The below has taken from another sequence of the same stream.
 
$ TZ=Asia/Kolkata tshark -T fields -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e tcp.time_delta -e frame.len -e _ws.col.Info -tad -Y tcp.seq==793042 -r 0360-vxlan_sys_4789_scposw1075.ocpprd01.prd.isaq.app_23_04_2024-20_59_33-UTC.pcap Apr 24, 2024 02:30:54.252911000 IST 10.131.1.163 53884 172.16.97.16 60728 0.000394000 243 OPNQRY | SQLDTA

$ TZ=Asia/Kolkata tshark -T fields -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e tcp.time_delta -e frame.len -e _ws.col.Info -tad -Y tcp.seq==793042 -r 0400-vxlan_sys_4789_scposi1067.ocpprd01.prd.isaq.app_23_04_2024-20_59_48-UTC.pcap Apr 24, 2024 02:30:54.253077000 IST 10.131.1.163 53884 172.16.97.16 60728 0.001008000 243 OPNQRY | SQLDTA