Summary: ================ Pod to pod [east-west] HW Offload Pass. Pod to external [north-south] HW offload is not working. Pod to nodePort service [north-south] Depending on test HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957. Depending on test HW offload is not working. Pod to clusterIP service Backed by regular pod (same node) [east-west] HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by regular pod (diff node) [east-west] HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by hostnetwork pod (same node) [north-south] HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by hostnetwork pod (diff node) [north-south] HW offload is not working. Hostnetwork pod to ClusterIP service Backed by regular pod (diff node) [north-south] HW offload is not working. Also affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by regular pod (same node) [north-south] HW Offload Pass. Backed by hostnetwork pod (diff node) [north-south] HW offload is not working. Backed by hostnetwork pod (same node) [north-south] HW offload is not working. Hostnetwork pod to NodePort service [north-south] Depending on test HW offload is not working. Also affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957. External network traffic to nodePort/External IP service [north-south] HW offload is not working. External network traffic to pods (multiple external gateway) [north-south] Unsure how to test this. Bare metal hosts running this kernel version 8.6. ==================================================== Linux worker-advnetlab23 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux Linux worker-advnetlab24 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux Environment Variables: ======================== export FT_NAMESPACE="default" export FT_REQ_SERVER_NODE=worker-advnetlab23 export FT_REQ_REMOTE_CLIENT_NODE=worker-advnetlab24 export FT_SRIOV_NODE_LABEL="feature.node.kubernetes.io/network-sriov.capable" export SRIOV_RESOURCE_NAME="openshift.io/mlxnics" export TEST_IMAGE="quay.io/wizhao/ft-base-image:0.8-x86_64" Default/Override Values: Launch Control: FT_HOSTONLY false FT_CLIENTONLY false FT_NAMESPACE default FT_REQ_SERVER_NODE worker-advnetlab23 FT_REQ_REMOTE_CLIENT_NODE worker-advnetlab24 FT_SRIOV_NODE_LABEL feature.node.kubernetes.io/network-sriov.capable FT_EXPORT_SVC false Label Management: FT_SERVER_NODE_LABEL ft.ServerPod FT_CLIENT_NODE_LABEL ft.ClientPod Test Control: TEST_CASE (0 means all) 1 VERBOSE false FT_VARS true FT_NOTES false FT_DEBUG false CURL true CURL_CMD curl -m 5 IPERF true IPERF_CMD iperf3 IPERF_TIME 10 FT_CLIENT_CPU_MASK OVN_TRACE false OVN_TRACE_CMD ./ovnkube-trace -loglevel=5 -tcp FT_SVC_QUALIFIER FT_MC_NAMESPACE submariner-operator FT_MC_CO_SERVER_LABEL submariner.io/gateway=true OVN Trace Control: OVN_K_NAMESPACE ovn-kubernetes SSL_ENABLE -noSSL From YAML Files: NET_ATTACH_DEF_NAME ftnetattach SRIOV_RESOURCE_NAME openshift.io/mlxnics TEST_IMAGE quay.io/wizhao/ft-base-image:0.8-x86_64 CLIENT_POD_NAME_PREFIX ft-client-pod http Server: HTTP_SERVER_POD_NAME ft-http-server-pod-v4 HTTP_SERVER_HOST_POD_NAME ft-http-server-host-v4 HTTP_CLUSTERIP_POD_SVC_NAME ft-http-service-clusterip-pod-v4 HTTP_CLUSTERIP_POD_SVC_PORT 8080 HTTP_CLUSTERIP_HOST_SVC_NAME ft-http-service-clusterip-host-v4 HTTP_CLUSTERIP_HOST_SVC_PORT 8079 HTTP_NODEPORT_SVC_NAME ft-http-service-nodeport-pod-v4 HTTP_NODEPORT_POD_SVC_PORT 30080 HTTP_NODEPORT_HOST_SVC_NAME ft-http-service-nodeport-host-v4 HTTP_NODEPORT_HOST_SVC_PORT 30079 iperf Server: IPERF_SERVER_POD_NAME ft-iperf-server-pod-v4 IPERF_SERVER_HOST_POD_NAME ft-iperf-server-host-v4 IPERF_CLUSTERIP_POD_SVC_NAME ft-iperf-service-clusterip-pod-v4 IPERF_CLUSTERIP_POD_SVC_PORT 5201 IPERF_CLUSTERIP_HOST_SVC_NAME ft-iperf-service-clusterip-host-v4 IPERF_CLUSTERIP_HOST_SVC_PORT 5202 IPERF_NODEPORT_POD_SVC_NAME ft-iperf-service-nodeport-pod-v4 IPERF_NODEPORT_POD_SVC_PORT 30201 IPERF_NODEPORT_HOST_SVC_NAME ft-iperf-service-nodeport-host-v4 IPERF_NODEPORT_HOST_SVC_PORT 30202 SERVER_PATH /etc/httpserver/ POD_SERVER_STRING Server - Pod Backend Reached HOST_SERVER_STRING Server - Host Backend Reached EXTERNAL_SERVER_STRING The document has moved KUBEAPI_SERVER_STRING serverAddressByClientCIDRs External Access: EXTERNAL_IP 8.8.8.8 EXTERNAL_URL google.com Queried Values: Pod Backed: HTTP_SERVER_POD_IP 10.131.0.32 IPERF_SERVER_POD_IP 10.131.0.33 SERVER_POD_NODE worker-advnetlab23 LOCAL_CLIENT_NODE worker-advnetlab23 LOCAL_CLIENT_POD ft-client-pod-sriov-5tvzh REMOTE_CLIENT_NODE_LIST worker-advnetlab24 REMOTE_CLIENT_POD_LIST ft-client-pod-sriov-h629l HTTP_CLUSTERIP_POD_SVC_IPV4_LIST 172.30.222.119 HTTP_CLUSTERIP_POD_SVC_PORT 8080 HTTP_NODEPORT_POD_SVC_IPV4_LIST 172.30.254.83 HTTP_NODEPORT_POD_SVC_PORT 30080 IPERF_CLUSTERIP_POD_SVC_IPV4_LIST 172.30.33.58 IPERF_CLUSTERIP_POD_SVC_PORT 5201 IPERF_NODEPORT_POD_SVC_IPV4_LIST 172.30.115.113 IPERF_NODEPORT_POD_SVC_PORT 30201 Host backed: HTTP_SERVER_HOST_IP 192.168.111.33 IPERF_SERVER_HOST_IP 192.168.111.33 SERVER_HOST_NODE worker-advnetlab23 LOCAL_CLIENT_HOST_NODE worker-advnetlab23 LOCAL_CLIENT_HOST_POD ft-client-pod-host-tznls REMOTE_CLIENT_HOST_NODE_LIST worker-advnetlab24 REMOTE_CLIENT_HOST_POD_LIST ft-client-pod-host-bmkr5 HTTP_CLUSTERIP_HOST_SVC_IPV4_LIST 172.30.235.16 HTTP_CLUSTERIP_HOST_SVC_PORT 8079 HTTP_NODEPORT_HOST_SVC_IPV4_LIST 172.30.96.63 HTTP_NODEPORT_HOST_SVC_PORT 30079 IPERF_CLUSTERIP_HOST_SVC_IPV4_LIST 172.30.244.122 IPERF_CLUSTERIP_HOST_SVC_PORT 5202 IPERF_NODEPORT_HOST_SVC_IPV4_LIST 172.30.34.186 IPERF_NODEPORT_HOST_SVC_PORT 30202 Kubernetes API: HTTP_CLUSTERIP_KUBEAPI_SVC_IPV4 172.30.0.1 HTTP_CLUSTERIP_KUBEAPI_SVC_PORT 443 HTTP_CLUSTERIP_KUBEAPI_EP_IP 192.168.111.20 HTTP_CLUSTERIP_KUBEAPI_EP_PORT 6443 HTTP_CLUSTERIP_KUBEAPI_SVC_NAME kubernetes.default.svc [root@wsfd-advnetlab25 ovn-kuber-traffic-flow-tests]# oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ft-client-pod-host-bmkr5 1/1 Running 0 17m 192.168.111.34 worker-advnetlab24 ft-client-pod-host-fj7h9 0/1 Evicted 0 48s master-0 ft-client-pod-host-h8b4m 1/1 Running 0 17m 192.168.111.21 master-1 ft-client-pod-host-pg26k 0/1 Evicted 0 48s master-2 ft-client-pod-host-tznls 1/1 Running 0 17m 192.168.111.33 worker-advnetlab23 ft-client-pod-sriov-5tvzh 1/1 Running 0 17m 10.131.0.34 worker-advnetlab23 ft-client-pod-sriov-h629l 1/1 Running 0 17m 10.128.2.48 worker-advnetlab24 ft-http-server-host-v4 1/1 Running 0 17m 192.168.111.33 worker-advnetlab23 ft-http-server-pod-v4 1/1 Running 0 17m 10.131.0.32 worker-advnetlab23 ft-iperf-server-host-v4 1/1 Running 0 17m 192.168.111.33 worker-advnetlab23 ft-iperf-server-pod-v4 1/1 Running 0 17m 10.131.0.33 worker-advnetlab23 [core@worker-advnetlab23 ~]$ sudo crictl ps --all CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD 2f22f650712a2 1cc889277705da9f81c238fbec5725813d25b50be0f9914f598055b7b55afeb6 13 minutes ago Exited collect-profiles 0 9a97429afe673 collect-profiles-27565635-s4zrc 3468efb49631c d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 18 minutes ago Running ft-iperf-server-pod-v4 0 2d85aede96ede ft-iperf-server-pod-v4 61194f93324b6 d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 18 minutes ago Running ft-http-server-pod-v4 0 2303b61900c3a ft-http-server-pod-v4 575656e861ba9 d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 18 minutes ago Running ft-client-pod 0 c614a423e5794 ft-client-pod-sriov-5tvzh b2f12e470631d d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 18 minutes ago Running ft-client-host 0 d3d4c8f3bdf36 ft-client-pod-host-tznls 5d48c0aa011fc d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 18 minutes ago Running ft-iperf-server-host-v4 0 4091c9c0f4458 ft-iperf-server-host-v4 eb8a3973a3eaf d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 18 minutes ago Running ft-http-server-host-node-v4 0 373b072dc65eb ft-http-server-host-v4 e82d57b93f2fc 1cc889277705da9f81c238fbec5725813d25b50be0f9914f598055b7b55afeb6 28 minutes ago Exited collect-profiles 0 b246a6816095d collect-profiles-27565620-qjj5h 523c49fc144d4 c31154a2596700d254d48507d2411d7d7b7669e8337fce0e8220b4a37c3baa91 41 minutes ago Running check-endpoints 0 99a686ea787ac network-check-source-74b5696f69-bpt6k a1534d444d6da 9207dace6755820e3b6c055f9d760a9454ab9266d12d2b918669316051915529 42 minutes ago Running sriov-device-plugin 0 936026a5116b1 sriov-device-plugin-9xpj2 5d70a88bda94f 09a9a7c0ccadb9dc48a82684f82d7b1f965bc815e3e0b51ec66e5b9e070de19b 42 minutes ago Running prometheus-adapter 0 e0400e49b5c21 prometheus-adapter-746bf6988-mhw7q 41be112eec4e6 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 42 minutes ago Running kube-rbac-proxy 3 4bbd305fe7602 dns-default-hhsqs 85fec145eed67 67aeaf5aed896447f3989a480d6de2f9d332c4871ecf01e2e37a35e60e997f4f 42 minutes ago Running kube-multus-additional-cni-plugins 3 843270f99a4e1 multus-additional-cni-plugins-gbddq 222715270f1e0 af4e4fe2f08171b443d115b96009c302d5fd8dd745ae90b6c97cede59151cffc 42 minutes ago Running dns 3 4bbd305fe7602 dns-default-hhsqs 98a1d443b0bd4 bd5a53f5dbb4b0e5242cc23225175c5863ea0eea5485bb4ac3941099523ddcf0 42 minutes ago Running serve-healthcheck-canary 3 b632a5af225a9 ingress-canary-bjwbd c116b491aea41 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 42 minutes ago Running kube-rbac-proxy 3 1c5f0d74547a7 network-metrics-daemon-kscnr 82a8520e2fa8f da506d6390c73a691b6ba6987705d09cbce969f41b2c2d162515d3f2fcb368d3 42 minutes ago Exited whereabouts-cni 3 843270f99a4e1 multus-additional-cni-plugins-gbddq 7751d3cfd21b6 c31154a2596700d254d48507d2411d7d7b7669e8337fce0e8220b4a37c3baa91 42 minutes ago Running network-check-target-container 3 9db20cfe02046 network-check-target-hdpz4 5dda0ceb0db23 031d5bc2fb191eca29a56160538b050a99d10547709c5e57b572a7e7928e802c 42 minutes ago Running network-metrics-daemon 3 1c5f0d74547a7 network-metrics-daemon-kscnr f27e7ed2cdb08 da506d6390c73a691b6ba6987705d09cbce969f41b2c2d162515d3f2fcb368d3 42 minutes ago Exited whereabouts-cni-bincopy 3 843270f99a4e1 multus-additional-cni-plugins-gbddq b65f85fb85ae4 f1bd50bc378e43abe884d9503c38dc439beba6e3133a36ba32690ab7fc861067 42 minutes ago Exited routeoverride-cni 3 843270f99a4e1 multus-additional-cni-plugins-gbddq 287f4b58222a0 1b94f6411ce474c33296c5abe81bfaa1cfe1d9a8ef6ba72411bcdacd0dd9b2d5 42 minutes ago Exited bond-cni-plugin 3 843270f99a4e1 multus-additional-cni-plugins-gbddq 9934f65fe7288 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 42 minutes ago Running kube-rbac-proxy 3 f9b5d8169cf6f node-exporter-m98dq 69d88af85dd39 0fbc0f72d2c4ef55a7ee7c76eec8fd4cdd01785587b35c1fce0254843d636bde 42 minutes ago Running node-exporter 3 f9b5d8169cf6f node-exporter-m98dq 9e683ad87a433 c6ab9d7050a41d41f6cfd93db921060c614665bcfd8280d0c766b8c8ccf54e91 42 minutes ago Exited cni-plugins 3 843270f99a4e1 multus-additional-cni-plugins-gbddq 51221a0499fba 1467ec4386c135a6c7cd8c96674d9ff0eb472e3616cad68002accc203a9798d9 42 minutes ago Running ovnkube-node 3 6947abf9bcb2b ovnkube-node-f8wv8 596c50f726feb 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 42 minutes ago Running kube-rbac-proxy-ovn-metrics 3 6947abf9bcb2b ovnkube-node-f8wv8 3724c11f1c19c 6de7db07b3b6abd782ec56ab210ed8093d3c4b8d0301d6da5f5ce7abbfcc5e38 42 minutes ago Running sriov-network-config-daemon 3 8282419a75e67 sriov-network-config-daemon-s96p5 82a60e709c938 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 42 minutes ago Running kube-rbac-proxy 3 6947abf9bcb2b ovnkube-node-f8wv8 cb8f29f10c37a fb08eea2866f547a005404160e41d73f5ae4f4086115d87bf15a5faa7763f803 42 minutes ago Running sriov-infiniband-cni 3 8282419a75e67 sriov-network-config-daemon-s96p5 3e41cc9ba861e 1467ec4386c135a6c7cd8c96674d9ff0eb472e3616cad68002accc203a9798d9 42 minutes ago Running ovn-acl-logging 3 6947abf9bcb2b ovnkube-node-f8wv8 bb495ccd97188 af697a7f26d3e05c5ac2599bc4bb3cc48911ed73f1acbd46482eeae98e9e1f6c 42 minutes ago Running oauth-proxy 3 58c7f9632af2f machine-config-daemon-lq9cw 36df77b8c3e88 decbb9c7c0a90efa76fbaf12fda0b1f5b3900331d480c4190132c6a04086756d 42 minutes ago Running sriov-cni 3 8282419a75e67 sriov-network-config-daemon-s96p5 41bc5f9cda4c2 1467ec4386c135a6c7cd8c96674d9ff0eb472e3616cad68002accc203a9798d9 42 minutes ago Running ovn-controller 3 6947abf9bcb2b ovnkube-node-f8wv8 4555fad8b7a3d 214a636d5b6c7c13ce55ccc813af43d69f25ba16d03f51ba46173df44973357c 42 minutes ago Running dns-node-resolver 3 6fb614c685ce2 node-resolver-bf5f7 860f03733a946 691c0da27531b745c45f0234ad9121b83da520817f64b46f7877572242139587 42 minutes ago Running node-ca 3 f522719b9720f node-ca-rsfxl 4de80143f72fb f0f68c5033524331a8b5acb168482d21430db9e98c1ddbe591626209233a210d 42 minutes ago Exited egress-router-binary-copy 3 843270f99a4e1 multus-additional-cni-plugins-gbddq 0b98a92f6b3ea 67aeaf5aed896447f3989a480d6de2f9d332c4871ecf01e2e37a35e60e997f4f 42 minutes ago Running kube-multus 3 81c5b62a1995b multus-sxvk5 902f332410078 0fbc0f72d2c4ef55a7ee7c76eec8fd4cdd01785587b35c1fce0254843d636bde 42 minutes ago Exited init-textfile 3 f9b5d8169cf6f node-exporter-m98dq 2e0f876a6c5df 1afdc03070caf45f5deed5799fc848850c56ea0c3ebe1e14e7f160f2c47e8085 42 minutes ago Running machine-config-daemon 3 58c7f9632af2f machine-config-daemon-lq9cw 8498bf08268f2 2d193cb7261e61aa2c869d5f3c1a6d8bcb173931b87566456b63aff1caa4f0e2 42 minutes ago Running tuned 3 ab79eea826a49 tuned-ckf2k ab7833a1d02d8 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 42 minutes ago Running coredns-monitor 3 bc164caccb6ce coredns-worker-advnetlab23 5260ca2623985 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 42 minutes ago Running keepalived-monitor 3 05d3918b4d7df keepalived-worker-advnetlab23 03e5b57e02920 ec95d944d92e30709bc1a0a68f618acb1ed442133ea91c14fb27ad759f143e2f 42 minutes ago Running keepalived 3 05d3918b4d7df keepalived-worker-advnetlab23 9302d8661fba9 af4e4fe2f08171b443d115b96009c302d5fd8dd745ae90b6c97cede59151cffc 42 minutes ago Running coredns 3 bc164caccb6ce coredns-worker-advnetlab23 90fb1e5eca5a5 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 42 minutes ago Exited render-config-keepalived 3 05d3918b4d7df keepalived-worker-advnetlab23 2a9c31eff538d 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 42 minutes ago Exited render-config-coredns 3 bc164caccb6ce coredns-worker-advnetlab23 [core@worker-advnetlab23 ~]$ sudo crictl ps --all | grep ft- 3468efb49631c d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 19 minutes ago Running ft-iperf-server-pod-v4 0 2d85aede96ede ft-iperf-server-pod-v4 61194f93324b6 d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 19 minutes ago Running ft-http-server-pod-v4 0 2303b61900c3a ft-http-server-pod-v4 575656e861ba9 d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 19 minutes ago Running ft-client-pod 0 c614a423e5794 ft-client-pod-sriov-5tvzh b2f12e470631d d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 19 minutes ago Running ft-client-host 0 d3d4c8f3bdf36 ft-client-pod-host-tznls 5d48c0aa011fc d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 19 minutes ago Running ft-iperf-server-host-v4 0 4091c9c0f4458 ft-iperf-server-host-v4 eb8a3973a3eaf d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 19 minutes ago Running ft-http-server-host-node-v4 0 373b072dc65eb ft-http-server-host-v4 [core@worker-advnetlab23 ~]$ sudo ovs-vsctl show | grep 2d85aede96ede Port "2d85aede96ede73" Interface "2d85aede96ede73" [core@worker-advnetlab23 ~]$ sudo ovs-vsctl show | grep 2303b61900c3a Port "2303b61900c3ab2" Interface "2303b61900c3ab2" [core@worker-advnetlab23 ~]$ sudo ovs-vsctl show | grep c614a423e5794 Port c614a423e5794cb Interface c614a423e5794cb [core@worker-advnetlab23 ~]$ sudo ovs-vsctl show | grep d3d4c8f3bdf36 [core@worker-advnetlab23 ~]$ sudo ovs-vsctl show | grep 4091c9c0f4458 [core@worker-advnetlab23 ~]$ sudo ovs-vsctl show | grep 373b072dc65eb [core@worker-advnetlab24 ~]$ sudo crictl ps --all CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD 69bb410a62baf d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 22 minutes ago Running ft-client-host 0 6845eba58af02 ft-client-pod-host-bmkr5 75229e0b7bad8 d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 22 minutes ago Running ft-client-pod 0 e29d580ccf1c6 ft-client-pod-sriov-h629l ec11fb12b5f06 9207dace6755820e3b6c055f9d760a9454ab9266d12d2b918669316051915529 39 minutes ago Running sriov-device-plugin 0 fd94c9e784a0f sriov-device-plugin-7bnlc b459cead392a0 67aeaf5aed896447f3989a480d6de2f9d332c4871ecf01e2e37a35e60e997f4f 40 minutes ago Running kube-multus-additional-cni-plugins 3 311c571130268 multus-additional-cni-plugins-l99s5 798de142b6ef1 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 40 minutes ago Running kube-rbac-proxy 3 df752d708b6d7 network-metrics-daemon-v6xsh 53792d04ff9f9 bd5a53f5dbb4b0e5242cc23225175c5863ea0eea5485bb4ac3941099523ddcf0 40 minutes ago Running serve-healthcheck-canary 3 927e7a83be17a ingress-canary-stfrz eafda084937c4 031d5bc2fb191eca29a56160538b050a99d10547709c5e57b572a7e7928e802c 40 minutes ago Running network-metrics-daemon 3 df752d708b6d7 network-metrics-daemon-v6xsh 7cac1ec97dc95 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 40 minutes ago Running kube-rbac-proxy 3 65da9bc1909df dns-default-4284j 1fddc4b8678b3 c31154a2596700d254d48507d2411d7d7b7669e8337fce0e8220b4a37c3baa91 40 minutes ago Running network-check-target-container 3 d127c645596f3 network-check-target-slnx6 485e1f349b1f9 af4e4fe2f08171b443d115b96009c302d5fd8dd745ae90b6c97cede59151cffc 40 minutes ago Running dns 3 65da9bc1909df dns-default-4284j 117947dfe57b2 da506d6390c73a691b6ba6987705d09cbce969f41b2c2d162515d3f2fcb368d3 40 minutes ago Exited whereabouts-cni 3 311c571130268 multus-additional-cni-plugins-l99s5 404b4743d9917 da506d6390c73a691b6ba6987705d09cbce969f41b2c2d162515d3f2fcb368d3 40 minutes ago Exited whereabouts-cni-bincopy 3 311c571130268 multus-additional-cni-plugins-l99s5 557932d0eb14e f1bd50bc378e43abe884d9503c38dc439beba6e3133a36ba32690ab7fc861067 40 minutes ago Exited routeoverride-cni 3 311c571130268 multus-additional-cni-plugins-l99s5 315d2480ffaf3 1b94f6411ce474c33296c5abe81bfaa1cfe1d9a8ef6ba72411bcdacd0dd9b2d5 40 minutes ago Exited bond-cni-plugin 3 311c571130268 multus-additional-cni-plugins-l99s5 2ca0cd9fac2fa 1467ec4386c135a6c7cd8c96674d9ff0eb472e3616cad68002accc203a9798d9 40 minutes ago Running ovnkube-node 3 7c781e476d8e1 ovnkube-node-67xp4 192b48927dfcf 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 40 minutes ago Running kube-rbac-proxy-ovn-metrics 3 7c781e476d8e1 ovnkube-node-67xp4 4367ad128acdf 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 40 minutes ago Running kube-rbac-proxy 3 c8adc81c83f0d node-exporter-rg4zl f67989aa96f84 6de7db07b3b6abd782ec56ab210ed8093d3c4b8d0301d6da5f5ce7abbfcc5e38 40 minutes ago Running sriov-network-config-daemon 3 6f3cfab3d2c1f sriov-network-config-daemon-z7ksv 6234b253ff022 336cc515647a136b4978b5c858fd616b796ada0dad1a4612508487c54c360b81 40 minutes ago Running kube-rbac-proxy 3 7c781e476d8e1 ovnkube-node-67xp4 df96ef8f55fd2 0fbc0f72d2c4ef55a7ee7c76eec8fd4cdd01785587b35c1fce0254843d636bde 40 minutes ago Running node-exporter 3 c8adc81c83f0d node-exporter-rg4zl 9e24ac431b8c0 c6ab9d7050a41d41f6cfd93db921060c614665bcfd8280d0c766b8c8ccf54e91 40 minutes ago Exited cni-plugins 3 311c571130268 multus-additional-cni-plugins-l99s5 1a5ef944f0436 fb08eea2866f547a005404160e41d73f5ae4f4086115d87bf15a5faa7763f803 40 minutes ago Running sriov-infiniband-cni 3 6f3cfab3d2c1f sriov-network-config-daemon-z7ksv 7cada1503a0c5 1467ec4386c135a6c7cd8c96674d9ff0eb472e3616cad68002accc203a9798d9 40 minutes ago Running ovn-acl-logging 3 7c781e476d8e1 ovnkube-node-67xp4 e9378caa3af53 af697a7f26d3e05c5ac2599bc4bb3cc48911ed73f1acbd46482eeae98e9e1f6c 40 minutes ago Running oauth-proxy 3 00327fa5249ca machine-config-daemon-tjlpq 0cf5a2d554bb4 2d193cb7261e61aa2c869d5f3c1a6d8bcb173931b87566456b63aff1caa4f0e2 40 minutes ago Running tuned 3 027c9080ca2eb tuned-nrhtk 943d44afb6610 214a636d5b6c7c13ce55ccc813af43d69f25ba16d03f51ba46173df44973357c 40 minutes ago Running dns-node-resolver 3 11ca52f29dab8 node-resolver-9zkdd cc2220604ee6f 67aeaf5aed896447f3989a480d6de2f9d332c4871ecf01e2e37a35e60e997f4f 40 minutes ago Running kube-multus 3 6ddd46e59d68c multus-rlsl5 58f6c8c79c3d5 691c0da27531b745c45f0234ad9121b83da520817f64b46f7877572242139587 40 minutes ago Running node-ca 3 664d26b25b61d node-ca-9c8g9 b84d0c3902f79 decbb9c7c0a90efa76fbaf12fda0b1f5b3900331d480c4190132c6a04086756d 40 minutes ago Running sriov-cni 3 6f3cfab3d2c1f sriov-network-config-daemon-z7ksv 13acb3ff9cd3b 1467ec4386c135a6c7cd8c96674d9ff0eb472e3616cad68002accc203a9798d9 40 minutes ago Running ovn-controller 3 7c781e476d8e1 ovnkube-node-67xp4 1794b2a9241ef 0fbc0f72d2c4ef55a7ee7c76eec8fd4cdd01785587b35c1fce0254843d636bde 40 minutes ago Exited init-textfile 3 c8adc81c83f0d node-exporter-rg4zl 2df19f7f451f4 1afdc03070caf45f5deed5799fc848850c56ea0c3ebe1e14e7f160f2c47e8085 40 minutes ago Running machine-config-daemon 3 00327fa5249ca machine-config-daemon-tjlpq fd76e6f33888d f0f68c5033524331a8b5acb168482d21430db9e98c1ddbe591626209233a210d 40 minutes ago Exited egress-router-binary-copy 3 311c571130268 multus-additional-cni-plugins-l99s5 23a9c872e633d 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 40 minutes ago Running keepalived-monitor 3 ffdc0b35320fd keepalived-worker-advnetlab24 f763b4beb6ffe 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 40 minutes ago Running coredns-monitor 3 f93f7e358b161 coredns-worker-advnetlab24 27bd8cb5a9800 af4e4fe2f08171b443d115b96009c302d5fd8dd745ae90b6c97cede59151cffc 40 minutes ago Running coredns 3 f93f7e358b161 coredns-worker-advnetlab24 f51b22ef85f4c ec95d944d92e30709bc1a0a68f618acb1ed442133ea91c14fb27ad759f143e2f 40 minutes ago Running keepalived 3 ffdc0b35320fd keepalived-worker-advnetlab24 40e1e62a76972 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 40 minutes ago Exited render-config-coredns 3 f93f7e358b161 coredns-worker-advnetlab24 b4386b8ef6359 8fba0add32eba9ca68b551409e406935820882149cec3bb0db2dcd28f50f16b0 40 minutes ago Exited render-config-keepalived 3 ffdc0b35320fd keepalived-worker-advnetlab24 [core@worker-advnetlab24 ~]$ sudo crictl ps --all | grep ft- 69bb410a62baf d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 22 minutes ago Running ft-client-host 0 6845eba58af02 ft-client-pod-host-bmkr5 75229e0b7bad8 d0121efcab7ad1a8c4be3a59b46716c423b595c4e609d93f7600e76ba98ba4cd 22 minutes ago Running ft-client-pod 0 e29d580ccf1c6 ft-client-pod-sriov-h629l [core@worker-advnetlab24 ~]$ sudo ovs-vsctl show | grep 6845eba58af02 [core@worker-advnetlab24 ~]$ sudo ovs-vsctl show | grep e29d580ccf1c6 Port e29d580ccf1c661 Interface e29d580ccf1c661 ========================= Pod to pod [east-west] ========================= TEST_CASE=1 FT_NOTES=false FT_VARS=true IPERF=true IPERF_TIME=50 ./test.sh FLOW 01: Pod to Pod traffic --------------------------- *** 1-a: Pod to Pod (Same Node) *** admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://10.131.0.32:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- iperf3 -c 10.131.0.33 -p 5201 -t 5 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-5.00 sec 15.5 GBytes 26.6 Gbits/sec 2607 sender [ 5] 0.00-5.00 sec 15.5 GBytes 26.6 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:0/s tx_packets:0/s vport_rx_packets:43225/s vport_tx_packets:2354560/s c614a423e5794cb: rx_packets:0/s tx_packets:0/s vport_rx_packets:2358147/s vport_tx_packets:43345/s ens2f0: rx_packets:69/s tx_packets:106/s rx_packets_phy:69/s tx_packets_phy:106/s ens2f1: rx_packets:0/s tx_packets:1/s rx_packets_phy:0/s tx_packets_phy:1/s tcpdump -i 2d85aede96ede73 => No traffic after a few seconds. tcpdump -i 43227e1ca5d2bbb => No traffic after a few seconds. CONCLUSION: HW offload is working as expected. *** 1-b: Pod to Pod (Different Node) *** admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://10.131.0.32:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- iperf3 -c 10.131.0.33 -p 5201 -t 5 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-5.00 sec 15.8 GBytes 27.1 Gbits/sec 4405 sender [ 5] 0.00-5.00 sec 15.8 GBytes 27.1 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:0/s tx_packets:0/s vport_rx_packets:54861/s vport_tx_packets:2674580/s ens2f0: rx_packets:18/s tx_packets:22/s rx_packets_phy:2697727/s tx_packets_phy:55235/s tcpdump -i 2d85aede96ede73 => No traffic after a few seconds. worker-advnetlab24 e29d580ccf1c661: rx_packets:0/s tx_packets:0/s vport_rx_packets:2782133/s vport_tx_packets:56738/s ens3f0: rx_packets:20/s tx_packets:22/s rx_packets_phy:56968/s tx_packets_phy:2793109/s tcpdump -i e29d580ccf1c661 => No traffic after a few seconds. CONCLUSION: HW offload is working as expected. =============================== Pod to external [north-south] =============================== TEST_CASE=13 FT_NOTES=false FT_VARS=true IPERF=true IPERF_TIME=20 ./test.sh FLOW 13: Cluster -> External Network ------------------------------------ *** 13-a: Pod -> External Network *** admin:worker-advnetlab24 -> External:External kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://google.com/" SUCCESS *** 13-b: Host -> External Network *** admin:worker-advnetlab24 -> External:External kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://google.com/" SUCCESS Additional Test: kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl "https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/internal/megafile--all-reduced.json" worker-advnetlab24 e29d580ccf1c661: rx_packets:252/s tx_packets:3201/s vport_rx_packets:252/s vport_tx_packets:3201/s ens3f0: rx_packets:3468/s tx_packets:4118/s rx_packets_phy:3468/s tx_packets_phy:4118/s CONCLUSION: HW Offload working is inconclusive. It seems that some packets are received hardware offloaded. Additional Test Scenario: iperf3 -s -p 9999 on worker-advnetlab25 [root@wsfd-advnetlab25 ovn-kuber-traffic-flow-tests]# kubectl exec -it -n default ft-client-pod-sriov-h629l -- ping wsfd-advnetlab25.anl.lab.eng.bos.redhat.com PING wsfd-advnetlab25.anl.lab.eng.bos.redhat.com (10.19.128.53): 56 data bytes 64 bytes from 10.19.128.53: seq=0 ttl=42 time=149.072 ms 64 bytes from 10.19.128.53: seq=1 ttl=42 time=156.045 ms kubectl exec -it -n default ft-client-pod-sriov-nt6c6 -- iperf3 -c wsfd-advnetlab25.anl.lab.eng.bos.redhat.com -p 9999 -t 30 Connecting to host wsfd-advnetlab25.anl.lab.eng.bos.redhat.com, port 9999 [ 5] local 10.128.2.48 port 34516 connected to 10.19.128.53 port 9999 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 18.1 GBytes 5.18 Gbits/sec 965 sender [ 5] 0.00-30.04 sec 18.1 GBytes 5.17 Gbits/sec receiver worker-advnetlab24 e29d580ccf1c661: rx_packets:465896/s tx_packets:10228/s vport_rx_packets:465951/s vport_tx_packets:10228/s ens3f0: rx_packets:10307/s tx_packets:466550/s rx_packets_phy:10306/s tx_packets_phy:466558/s tcpdump -i e29d580ccf1c661 20:08:40.605722 IP worker-advnetlab24.35320 > wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct: Flags [.], seq 1860734016:1860798720, ack 1, win 213, options [nop,nop,TS val 4135748498 ecr 512808785], length 64704 20:08:40.605741 IP worker-advnetlab24.35320 > wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct: Flags [.], seq 1860798720:1860822984, ack 1, win 213, options [nop,nop,TS val 4135748498 ecr 512808785], length 24264 20:08:40.605800 IP worker-advnetlab24.35320 > wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct: Flags [.], seq 1860822984:1860887688, ack 1, win 213, options [nop,nop,TS val 4135748498 ecr 512808786], length 64704 20:08:40.605869 IP worker-advnetlab24.35320 > wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct: Flags [.], seq 1860887688:1860952392, ack 1, win 213, options [nop,nop,TS val 4135748498 ecr 512808786], length 64704 20:08:40.606329 IP wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct > 10.128.2.48.35320: Flags [.], ack 1860189462, win 24576, options [nop,nop,TS val 512808786 ecr 4135748498], length 0 20:08:40.606334 IP wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct > 10.128.2.48.35320: Flags [.], ack 1860254166, win 24576, options [nop,nop,TS val 512808786 ecr 4135748498], length 0 20:08:40.606336 IP wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct > 10.128.2.48.35320: Flags [.], ack 1860318870, win 24576, options [nop,nop,TS val 512808786 ecr 4135748498], length 0 20:08:40.606337 IP wsfd-advnetlab25.anl.lab.eng.bos.redhat.com.distinct > 10.128.2.48.35320: Flags [.], ack 1860383574, win 24576, options [nop,nop,TS val 512808786 ecr 4135748498], length 0 CONCLUSION: HW Offload working is not working. ====================================== Pod to nodePort service [north-south] ====================================== TEST_CASE=5 FT_NOTES=false FT_VARS=true IPERF=true IPERF_TIME=20 ./test.sh FLOW 05: Pod -> NodePort Service traffic (Pod Backend) ------------------------------------------------------ *** 5-a: Pod -> NodePort Service traffic (Pod Backend - Same Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://172.30.254.83:8080/etc/httpserver/" SUCCESS curl HostIP:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://192.168.111.33:30080/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://ft-http-service-nodeport-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- iperf3 -c 192.168.111.33 -p 30201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 51.8 GBytes 22.2 Gbits/sec 5477 sender [ 5] 0.00-20.00 sec 51.8 GBytes 22.2 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:41073/s tx_packets:0/s vport_rx_packets:41073/s vport_tx_packets:2254684/s c614a423e5794cb: rx_packets:0/s tx_packets:41048/s vport_rx_packets:2253692/s vport_tx_packets:41049/s ens2f0: rx_packets:44/s tx_packets:104/s rx_packets_phy:44/s tx_packets_phy:104/s tcpdump -i 2d85aede96ede73 20:13:56.145312 IP worker-advnetlab23.30201 > 10.131.0.34.59734: Flags [.], ack 3499037765, win 24576, options [nop,nop,TS val 4288493578 ecr 1644361899], length 0 tcpdump -i c614a423e5794cb 20:14:27.173831 IP worker-advnetlab23.30201 > 10.131.0.34.60076: Flags [.], ack 3172429050, win 3750, options [nop,nop,TS val 4288524607 ecr 1644392928], length 0 CONCLUSION: HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 *** 5-b: Pod -> NodePort Service traffic (Pod Backend - Different Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://172.30.254.83:8080/etc/httpserver/" SUCCESS curl HostIP:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://192.168.111.33:30080/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://ft-http-service-nodeport-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- iperf3 -c 192.168.111.33 -p 30201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 307 MBytes 129 Mbits/sec 3492 sender [ 5] 0.00-20.02 sec 306 MBytes 128 Mbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:0/s tx_packets:11385/s vport_rx_packets:731/s vport_tx_packets:11385/s ens2f0: rx_packets:11444/s tx_packets:51/s rx_packets_phy:11444/s tx_packets_phy:801/s tcpdump -i 2d85aede96ede73 20:18:01.386781 IP 100.64.0.5.49122 > 10.131.0.33.targus-getdata1: Flags [.], seq 154532062:154533410, ack 1, win 213, options [nop,nop,TS val 4101601911 ecr 1187758219], length 1348 20:18:01.386786 IP 100.64.0.5.49122 > 10.131.0.33.targus-getdata1: Flags [.], seq 154533410:154534758, ack 1, win 213, options [nop,nop,TS val 4101601911 ecr 1187758219], length 1348 20:18:01.386788 IP 100.64.0.5.49122 > 10.131.0.33.targus-getdata1: Flags [.], seq 154534758:154536106, ack 1, win 213, options [nop,nop,TS val 4101601911 ecr 1187758219], length 1348 20:18:01.386797 IP 100.64.0.5.49122 > 10.131.0.33.targus-getdata1: Flags [.], seq 154536106:154537454, ack 1, win 213, options [nop,nop,TS val 4101601911 ecr 1187758219], length 1348 20:18:01.386799 IP 100.64.0.5.49122 > 10.131.0.33.targus-getdata1: Flags [.], seq 154537454:154538802, ack 1, win 213, options [nop,nop,TS val 4101601911 ecr 1187758219], length 1348 worker-advnetlab24 e29d580ccf1c661: rx_packets:13353/s tx_packets:737/s vport_rx_packets:13353/s vport_tx_packets:737/s ens3f0: rx_packets:753/s tx_packets:13361/s rx_packets_phy:753/s tx_packets_phy:13361/s tcpdump -i e29d580ccf1c661 20:18:16.024491 IP worker-advnetlab24.49122 > worker-advnetlab23.30201: Flags [.], seq 394507028:394571732, ack 1, win 213, options [nop,nop,TS val 4101616571 ecr 1187772879], length 64704 20:18:16.024503 IP worker-advnetlab24.49122 > worker-advnetlab23.30201: Flags [.], seq 394571732:394574428, ack 1, win 213, options [nop,nop,TS val 4101616571 ecr 1187772879], length 2696 20:18:16.024534 IP worker-advnetlab24.49122 > worker-advnetlab23.30201: Flags [.], seq 394574428:394593300, ack 1, win 213, options [nop,nop,TS val 4101616571 ecr 1187772879], length 18872 20:18:16.029021 IP worker-advnetlab23.30201 > 10.128.2.48.49122: Flags [.], ack 394137714, win 21363, options [nop,nop,TS val 1187772883 ecr 4101616548], length 0 20:18:16.029033 IP worker-advnetlab23.30201 > 10.128.2.48.49122: Flags [.], ack 394144454, win 21400, options [nop,nop,TS val 1187772883 ecr 4101616548], length 0 20:18:16.029036 IP worker-advnetlab23.30201 > 10.128.2.48.49122: Flags [.], ack 394161978, win 21400, options [nop,nop,TS val 1187772883 ecr 4101616548], length 0 CONCLUSION: HW offload is not working. FLOW 06: Pod -> NodePort Service traffic (Host Backend) ------------------------------------------------------- *** 6-a: Pod -> NodePort Service traffic (Host Backend - Same Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://172.30.96.63:8079/etc/httpserver/" SUCCESS curl HostIP:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://192.168.111.33:30079/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://ft-http-service-nodeport-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- iperf3 -c 192.168.111.33 -p 30202 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 41.4 GBytes 17.8 Gbits/sec 5829 sender [ 5] 0.00-20.04 sec 41.4 GBytes 17.7 Gbits/sec receiver SUCCESS worker-advnetlab23 c614a423e5794cb: rx_packets:0/s tx_packets:32818/s vport_rx_packets:1623578/s vport_tx_packets:32818/s ens2f0: rx_packets:1619957/s tx_packets:102/s rx_packets_phy:57/s tx_packets_phy:102/s tcpdump -i c614a423e5794cb 20:38:42.027818 IP worker-advnetlab23.30202 > 10.131.0.34.53728: Flags [.], ack 2131150462, win 24225, options [nop,nop,TS val 3326983433 ecr 1645847782], length 0 CONCLUSION: HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 *** 6-b: Pod -> NodePort Service traffic (Host Backend - Different Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://172.30.96.63:8079/etc/httpserver/" SUCCESS curl HostIP:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://192.168.111.33:30079/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://ft-http-service-nodeport-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- iperf3 -c 192.168.111.33 -p 30202 -t 20 Summary (see iperf-logs/06-b-pod2nodePortSvc-hostBackend-diffNode.txt for full detail): [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 720 MBytes 302 Mbits/sec 2542 sender [ 5] 0.00-20.02 sec 717 MBytes 300 Mbits/sec receiver SUCCESS worker-advnetlab23 ens2f0: rx_packets:27758/s tx_packets:12886/s rx_packets_phy:27758/s tx_packets_phy:12883/s worker-advnetlab24 e29d580ccf1c661: rx_packets:24797/s tx_packets:11227/s vport_rx_packets:24796/s vport_tx_packets:11227/s ens3f0: rx_packets:11067/s tx_packets:24559/s rx_packets_phy:11067/s tx_packets_phy:24554/s tcpdump -i e29d580ccf1c661 20:40:07.357312 IP worker-advnetlab24.40946 > worker-advnetlab23.30202: Flags [.], seq 118800588:118839680, ack 1, win 213, options [nop,nop,TS val 4102927903 ecr 1979249940], length 39092 20:40:07.358218 IP worker-advnetlab23.30202 > 10.128.2.48.40946: Flags [.], ack 118272210, win 24576, options [nop,nop,TS val 1979249944 ecr 4102927876], length 0 20:40:07.358254 IP worker-advnetlab24.40946 > worker-advnetlab23.30202: Flags [.], seq 118839680:118865292, ack 1, win 213, options [nop,nop,TS val 4102927904 ecr 1979249944], length 25612 20:40:07.358646 IP worker-advnetlab23.30202 > 10.128.2.48.40946: Flags [.], ack 118274906, win 24576, options [nop,nop,TS val 1979249944 ecr 4102927880], length 0 20:40:07.358722 IP worker-advnetlab24.40946 > worker-advnetlab23.30202: Flags [.], seq 118865292:118878772, ack 1, win 213, options [nop,nop,TS val 4102927905 ecr 1979249944], length 13480 20:40:07.358950 IP worker-advnetlab23.30202 > 10.128.2.48.40946: Flags [.], ack 118278950, win 24576, options [nop,nop,TS val 1979249944 ecr 4102927880], length 0 20:40:07.358954 IP worker-advnetlab23.30202 > 10.128.2.48.40946: Flags [.], ack 118281646, win 24576, options [nop,nop,TS val 1979249944 ecr 4102927880], length 0 CONCLUSION: HW offload is not working. ====================================== Pod to clusterIP service ====================================== Backed by regular pod (same node) [east-west] FLOW 03: Pod -> Cluster IP Service traffic (Pod Backend) -------------------------------------------------------- *** 3-a: Pod -> Cluster IP Service traffic (Pod Backend - Same Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://172.30.222.119:8080/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://10.131.0.32:8080/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://ft-http-service-clusterip-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- iperf3 -c 172.30.33.58 -p 5201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 52.8 GBytes 22.7 Gbits/sec 7049 sender [ 5] 0.00-20.00 sec 52.8 GBytes 22.7 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:32116/s tx_packets:0/s vport_rx_packets:32116/s vport_tx_packets:1690844/s c614a423e5794cb: rx_packets:0/s tx_packets:32113/s vport_rx_packets:1691791/s vport_tx_packets:32113/s ens2f0: rx_packets:18/s tx_packets:15/s rx_packets_phy:18/s tx_packets_phy:15/s tcpdump -i 2d85aede96ede73 20:45:16.413321 IP 172.30.33.58.targus-getdata1 > 10.131.0.34.58384: Flags [.], ack 230518353, win 2072, options [nop,nop,TS val 4290373846 ecr 671647024], length 0 tcpdump -i c614a423e5794cb 20:45:32.899794 IP 172.30.33.58.targus-getdata1 > 10.131.0.34.58384: Flags [.], ack 478486081, win 24225, options [nop,nop,TS val 4290390333 ecr 671663511], length 0 CONCLUSION: HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by regular pod (diff node) [east-west] *** 3-b: Pod -> Cluster IP Service traffic (Pod Backend - Different Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://172.30.222.119:8080/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://10.131.0.32:8080/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://ft-http-service-clusterip-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- iperf3 -c 172.30.33.58 -p 5201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 59.6 GBytes 25.6 Gbits/sec 9542 sender [ 5] 0.00-20.00 sec 59.6 GBytes 25.6 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:0/s tx_packets:0/s vport_rx_packets:46866/s vport_tx_packets:2285289/s ens2f0: rx_packets:48/s tx_packets:24/s rx_packets_phy:2297129/s tx_packets_phy:47333/s tcpdump -i 2d85aede96ede73 => No traffic after a few seconds. worker-advnetlab24 e29d580ccf1c661: rx_packets:0/s tx_packets:46393/s vport_rx_packets:2311715/s vport_tx_packets:46395/s ens3f0: rx_packets:46703/s tx_packets:18/s rx_packets_phy:46702/s tx_packets_phy:2336805/s tcpdump -i e29d580ccf1c661 20:50:07.867824 IP 172.30.33.58.targus-getdata1 > 10.128.2.48.59966: Flags [.], ack 57986014, win 24225, options [nop,nop,TS val 1782139711 ecr 774036526], length 0 CONCLUSION: HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by hostnetwork pod (same node) [north-south] *** 4-a: Pod -> Cluster IP Service traffic (Host Backend - Same Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://172.30.235.16:8079/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://192.168.111.33:8079/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- curl -m 5 "http://ft-http-service-clusterip-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-5tvzh -- iperf3 -c 172.30.244.122 -p 5202 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 41.9 GBytes 18.0 Gbits/sec 7698 sender [ 5] 0.00-20.04 sec 41.9 GBytes 18.0 Gbits/sec receiver SUCCESS worker-advnetlab23 c614a423e5794cb: rx_packets:0/s tx_packets:34770/s vport_rx_packets:1649903/s vport_tx_packets:34770/s ens2f0: rx_packets:1649437/s tx_packets:36/s rx_packets_phy:49/s tx_packets_phy:36/s tcpdump -i c614a423e5794cb 20:55:36.915505 IP 172.30.244.122.targus-getdata2 > 10.131.0.34.34800: Flags [.], ack 2788691862, win 24225, options [nop,nop,TS val 3327998320 ecr 988361767], length 0 CONCLUSION: HW offload is partially working. Affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by hostnetwork pod (diff node) [north-south] *** 4-b: Pod -> Cluster IP Service traffic (Host Backend - Different Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://172.30.235.16:8079/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://192.168.111.33:8079/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- curl -m 5 "http://ft-http-service-clusterip-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-sriov-h629l -- iperf3 -c 172.30.244.122 -p 5202 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 11.5 GBytes 4.95 Gbits/sec 622 sender [ 5] 0.00-20.04 sec 11.5 GBytes 4.94 Gbits/sec receiver SUCCESS worker-advnetlab23 ens2f0: rx_packets:479012/s tx_packets:10489/s rx_packets_phy:478936/s tx_packets_phy:10487/s worker-advnetlab24 e29d580ccf1c661: rx_packets:502872/s tx_packets:10905/s vport_rx_packets:502938/s vport_tx_packets:10905/s ens3f0: rx_packets:10921/s tx_packets:502613/s rx_packets_phy:10923/s tx_packets_phy:502645/s tcpdump -i e29d580ccf1c661 20:57:01.678397 IP worker-advnetlab24.48222 > worker-advnetlab23.targus-getdata2: Flags [.], seq 2684868216:2684932920, ack 1, win 213, options [nop,nop,TS val 4066403778 ecr 3947053291], length 64704 20:57:01.678430 IP worker-advnetlab24.48222 > worker-advnetlab23.targus-getdata2: Flags [.], seq 2684932920:2684976056, ack 1, win 213, options [nop,nop,TS val 4066403778 ecr 3947053291], length 43136 20:57:01.678498 IP worker-advnetlab24.48222 > worker-advnetlab23.targus-getdata2: Flags [.], seq 2684976056:2685040760, ack 1, win 213, options [nop,nop,TS val 4066403778 ecr 3947053291], length 64704 20:57:01.678542 IP worker-advnetlab24.48222 > worker-advnetlab23.targus-getdata2: Flags [.], seq 2685040760:2685082548, ack 1, win 213, options [nop,nop,TS val 4066403778 ecr 3947053291], length 41788 20:57:01.678606 IP 172.30.244.122.targus-getdata2 > 10.128.2.48.48222: Flags [.], ack 2684350622, win 24576, options [nop,nop,TS val 3947053292 ecr 4066403778], length 0 20:57:01.678608 IP 172.30.244.122.targus-getdata2 > 10.128.2.48.48222: Flags [.], ack 2684415326, win 24576, options [nop,nop,TS val 3947053292 ecr 4066403778], length 0 20:57:01.678610 IP 172.30.244.122.targus-getdata2 > 10.128.2.48.48222: Flags [.], ack 2684480030, win 24576, options [nop,nop,TS val 3947053292 ecr 4066403778], length 0 20:57:01.678612 IP 172.30.244.122.targus-getdata2 > 10.128.2.48.48222: Flags [.], ack 2684544734, win 24576, options [nop,nop,TS val 3947053292 ecr 4066403778], length 0 CONCLUSION: HW offload is not working. ====================================== Hostnetwork pod to ClusterIP service ====================================== Backed by regular pod (same node) [north-south] *** 9-a: Host Pod -> Cluster IP Service traffic (Pod Backend - Same Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://172.30.222.119:8080/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://10.131.0.32:8080/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://ft-http-service-clusterip-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- iperf3 -c 172.30.33.58 -p 5201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 48.3 GBytes 20.7 Gbits/sec 2316 sender [ 5] 0.00-20.00 sec 48.3 GBytes 20.7 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:39491/s tx_packets:1833749/s vport_rx_packets:39491/s vport_tx_packets:1833749/s ens2f0: rx_packets:47/s tx_packets:38/s rx_packets_phy:47/s tx_packets_phy:38/s tcpdump -i 2d85aede96ede73 21:02:37.927794 IP 100.64.0.5.50954 > 10.131.0.33.targus-getdata1: Flags [.], seq 3521530918:3521595622, ack 1, win 213, options [nop,nop,TS val 4047617000 ecr 1190434783], length 64704 21:02:37.927818 IP 100.64.0.5.50954 > 10.131.0.33.targus-getdata1: Flags [.], seq 3521595622:3521660326, ack 1, win 213, options [nop,nop,TS val 4047617000 ecr 1190434783], length 64704 21:02:37.927834 IP 100.64.0.5.50954 > 10.131.0.33.targus-getdata1: Flags [P.], seq 3521660326:3521662306, ack 1, win 213, options [nop,nop,TS val 4047617000 ecr 1190434783], length 1980 21:02:37.927844 IP 172.30.33.58.targus-getdata1 > 169.254.169.2.50954: Flags [.], ack 3521526837, win 24576, options [nop,nop,TS val 1190434783 ecr 4047617000], length 0 21:02:37.927856 IP 100.64.0.5.50954 > 10.131.0.33.targus-getdata1: Flags [.], seq 3521662306:3521727010, ack 1, win 213, options [nop,nop,TS val 4047617000 ecr 1190434783], length 64704 21:02:37.927871 IP 172.30.33.58.targus-getdata1 > 169.254.169.2.50954: Flags [.], ack 3521591541, win 24576, options [nop,nop,TS val 1190434783 ecr 4047617000], length 0 CONCLUSION: HW offload is not working. Also affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 Backed by regular pod (diff node) [north-south] *** 9-b: Host Pod -> Cluster IP Service traffic (Pod Backend - Different Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://172.30.222.119:8080/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://10.131.0.32:8080/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://ft-http-service-clusterip-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- iperf3 -c 172.30.33.58 -p 5201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 18.9 GBytes 8.12 Gbits/sec 178 sender [ 5] 0.00-20.00 sec 18.9 GBytes 8.12 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:0/s tx_packets:0/s vport_rx_packets:19043/s vport_tx_packets:784167/s ens2f0: rx_packets:36/s tx_packets:21/s rx_packets_phy:786511/s tx_packets_phy:19493/s tcpdump -i => No traffic after a few seconds. worker-advnetlab24 ens3f0: rx_packets:19825/s tx_packets:769035/s rx_packets_phy:19828/s tx_packets_phy:769163/s CONCLUSION: HW offload is working as expected. Backed by hostnetwork pod (same node) [north-south] *** 10-a: Host Pod -> Cluster IP Service traffic (Host Backend - Same Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://172.30.235.16:8079/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://192.168.111.33:8079/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://ft-http-service-clusterip-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- iperf3 -c 172.30.244.122 -p 5202 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 1.82 GBytes 781 Mbits/sec 12316 sender [ 5] 0.00-20.04 sec 1.82 GBytes 778 Mbits/sec receiver SUCCESS worker-advnetlab23 ens2f0: rx_packets:81/s tx_packets:66/s rx_packets_phy:81/s tx_packets_phy:66/s CONCLUSION: HW offload is not working. Backed by hostnetwork pod (diff node) [north-south] *** 10-b: Host Pod -> Cluster IP Service traffic (Host Backend - Different Node) *** curl SvcClusterIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://172.30.235.16:8079/etc/httpserver/" SUCCESS curl SvcEndPointIP:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://192.168.111.33:8079/etc/httpserver/" SUCCESS curl SvcName:SvcPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://ft-http-service-clusterip-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- iperf3 -c 172.30.244.122 -p 5202 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 43.4 GBytes 18.7 Gbits/sec 7617 sender [ 5] 0.00-20.04 sec 43.4 GBytes 18.6 Gbits/sec receiver SUCCESS worker-advnetlab23 ens2f0: rx_packets:1731876/s tx_packets:36274/s rx_packets_phy:1732258/s tx_packets_phy:36279/s worker-advnetlab24 ens3f0: rx_packets:37296/s tx_packets:1782785/s rx_packets_phy:37297/s tx_packets_phy:1782814/s CONCLUSION: HW offload is not working. ====================================== Hostnetwork pod to NodePort service [north-south] ====================================== *** 11-a: Host Pod -> NodePort Service traffic (Pod Backend - Same Node) *** curl SvcClusterIP:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://172.30.254.83:8080/etc/httpserver/" SUCCESS curl hostIP:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://192.168.111.33:30080/etc/httpserver/" SUCCESS curl SvcName:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://ft-http-service-nodeport-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- iperf3 -c 192.168.111.33 -p 30201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 45.4 GBytes 19.5 Gbits/sec 3755 sender [ 5] 0.00-20.00 sec 45.4 GBytes 19.5 Gbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:40647/s tx_packets:1879509/s vport_rx_packets:40646/s vport_tx_packets:1879489/s ens2f0: rx_packets:19/s tx_packets:17/s rx_packets_phy:19/s tx_packets_phy:17/s tcpdump -i 2d85aede96ede73 21:48:36.828183 IP 100.64.0.5.53770 > 10.131.0.33.targus-getdata1: Flags [.], seq 2494260554:2494325258, ack 1, win 342, options [nop,nop,TS val 1240307537 ecr 1193193683], length 64704 21:48:36.828205 IP 100.64.0.5.53770 > 10.131.0.33.targus-getdata1: Flags [.], seq 2494325258:2494389962, ack 1, win 342, options [nop,nop,TS val 1240307537 ecr 1193193683], length 64704 21:48:36.828222 IP 100.64.0.5.53770 > 10.131.0.33.targus-getdata1: Flags [P.], seq 2494389962:2494392258, ack 1, win 342, options [nop,nop,TS val 1240307537 ecr 1193193683], length 2296 21:48:36.828236 IP 172.30.115.113.targus-getdata1 > 169.254.169.2.53770: Flags [.], ack 2493873641, win 24225, options [nop,nop,TS val 1193193683 ecr 1240307537], length 0 21:48:36.828247 IP 100.64.0.5.53770 > 10.131.0.33.targus-getdata1: Flags [.], seq 2494392258:2494456962, ack 1, win 342, options [nop,nop,TS val 1240307537 ecr 1193193683], length 64704 21:48:36.828262 IP 172.30.115.113.targus-getdata1 > 169.254.169.2.53770: Flags [.], ack 2493938345, win 24225, options [nop,nop,TS val 1193193683 ecr 1240307537], length 0 21:48:36.828269 IP 172.30.115.113.targus-getdata1 > 169.254.169.2.53770: Flags [.], ack 2494006693, win 24204, options [nop,nop,TS val 1193193683 ecr 1240307537], length 0 CONCLUSION: HW offload is not working. Also affected by bug: https://bugzilla.redhat.com/show_bug.cgi?id=2079957 *** 11-b: Host Pod -> NodePort Service traffic (Pod Backend - Different Node) *** curl SvcClusterIP:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://172.30.254.83:8080/etc/httpserver/" SUCCESS curl hostIP:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://192.168.111.33:30080/etc/httpserver/" SUCCESS curl SvcName:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://ft-http-service-nodeport-pod-v4:8080/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- iperf3 -c 192.168.111.33 -p 30201 -t 20 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 330 MBytes 139 Mbits/sec 3215 sender [ 5] 0.00-20.10 sec 329 MBytes 137 Mbits/sec receiver SUCCESS worker-advnetlab23 2d85aede96ede73: rx_packets:0/s tx_packets:12733/s vport_rx_packets:806/s vport_tx_packets:12733/s ens2f0: rx_packets:12957/s tx_packets:16/s rx_packets_phy:12953/s tx_packets_phy:813/s tcpdump -i 2d85aede96ede73 21:52:50.614564 IP 100.64.0.5.47168 > 10.131.0.33.targus-getdata1: Flags [.], seq 140527690:140529038, ack 1, win 229, options [nop,nop,TS val 1755287713 ecr 1193447455], length 1348 21:52:50.614574 IP 100.64.0.5.47168 > 10.131.0.33.targus-getdata1: Flags [.], seq 140529038:140530386, ack 1, win 229, options [nop,nop,TS val 1755287713 ecr 1193447455], length 1348 worker-advnetlab24 ens3f0: rx_packets:1146/s tx_packets:12776/s rx_packets_phy:1146/s tx_packets_phy:12776/s CONCLUSION: HW offload is not working. *** 12-a: Host Pod -> NodePort Service traffic (Host Backend - Same Node) *** curl SvcClusterIP:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://172.30.96.63:8079/etc/httpserver/" SUCCESS curl EndPointIP:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://192.168.111.33:30079/etc/httpserver/" SUCCESS curl SvcName:NODEPORT admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- curl -m 5 "http://ft-http-service-nodeport-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab23 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-tznls -- iperf3 -c 192.168.111.33 -p 30202 -t 20 Summary (see iperf-logs/12-a-host2nodePortSvc-hostBackend-sameNode.txt for full detail): [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 77.8 KBytes 31.9 Kbits/sec 0 sender [ 5] 0.00-20.04 sec 0.00 Bytes 0.00 bits/sec receiver SUCCESS worker-advnetlab23 ens2f0: rx_packets:64/s tx_packets:58/s rx_packets_phy:65/s tx_packets_phy:58/s CONCLUSION: HW offload is not working. *** 12-b: Host Pod -> NodePort Service traffic (Host Backend - Different Node) *** curl SvcClusterIP:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://172.30.96.63:8079/etc/httpserver/" SUCCESS curl EndPointIP:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://192.168.111.33:30079/etc/httpserver/" SUCCESS curl SvcName:NODEPORT admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- curl -m 5 "http://ft-http-service-nodeport-host-v4:8079/etc/httpserver/" SUCCESS admin:worker-advnetlab24 -> admin:worker-advnetlab23 kubectl exec -it -n default ft-client-pod-host-bmkr5 -- iperf3 -c 192.168.111.33 -p 30202 -t 20 command terminated with exit code 1 Summary (see iperf-logs/12-b-host2nodePortSvc-hostBackend-diffNode.txt for full detail): FAILED External network traffic to nodePort/External IP service [north-south] FLOW 15: External Network -> NodePort Service traffic ----------------------------------------------------- *** 15-a: External Network -> NodePort Service traffic (Pod Backend) *** curl EndPointIP:NODEPORT admin:External -> admin:worker-advnetlab23 curl -m 5 "http://192.168.111.33:30080/etc/httpserver/" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 164 100 164 0 0 353 0 --:--:-- --:--:-- --:--:-- 352 SUCCESS *** 15-b: External Network -> NodePort Service traffic (Host Backend) *** curl EndPointIP:NODEPORT admin:External -> admin:worker-advnetlab23 curl -m 5 "http://192.168.111.33:30079/etc/httpserver/" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 170 100 170 0 0 732 0 --:--:-- --:--:-- --:--:-- 732 SUCCESS Additional test: [root@wsfd-advnetlab25 ovn-kuber-traffic-flow-tests]# iperf3 -c 192.168.111.33 -p 30201 -t 30 Connecting to host 192.168.111.33, port 30201 [ 5] local 192.168.111.1 port 45882 connected to 192.168.111.33 port 30201 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 426 MBytes 119 Mbits/sec 5666 sender [ 5] 0.00-30.04 sec 423 MBytes 118 Mbits/sec receiver worker-advnetlab23 2d85aede96ede73: rx_packets:0/s tx_packets:13218/s vport_rx_packets:715/s vport_tx_packets:13216/s ens2f0: rx_packets:13130/s tx_packets:30/s rx_packets_phy:13130/s tx_packets_phy:741/s tcpdump -i 2d85aede96ede73 21:57:43.591001 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [P.], seq 110429546:110430894, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591003 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110430894:110432242, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591008 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110432242:110433590, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591011 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110433590:110434938, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591016 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110434938:110436286, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591018 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110436286:110437634, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591023 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110437634:110438982, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591026 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110438982:110440330, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591031 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110440330:110441678, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 21:57:43.591033 IP 100.64.0.5.45886 > 10.131.0.33.targus-getdata1: Flags [.], seq 110441678:110443026, ack 1, win 229, options [nop,nop,TS val 1186822555 ecr 1193740436], length 1348 CONCLUSION: HW offload is not working. External network traffic to pods (multiple external gateway) [north-south] Not Tested.