Story: https://issues.redhat.com/browse/SDN-3229 Test version: Client Version: 4.9.49 Server Version: 4.9.0-0.nightly-2022-10-10-022606 Kubernetes Version: v1.22.8+9e95cb9 Test steps: 1. oc get node [cloud-user@preserved-qiowang must-gather-test]$ oc get node NAME STATUS ROLES AGE VERSION qiowang-10103-57fmp-master-0.c.openshift-qe.internal Ready master 4h36m v1.22.8+9e95cb9 qiowang-10103-57fmp-master-1.c.openshift-qe.internal Ready master 4h36m v1.22.8+9e95cb9 qiowang-10103-57fmp-master-2.c.openshift-qe.internal Ready master 4h36m v1.22.8+9e95cb9 qiowang-10103-57fmp-worker-a-lmdtl.c.openshift-qe.internal Ready worker 4h26m v1.22.8+9e95cb9 qiowang-10103-57fmp-worker-b-lv5k8.c.openshift-qe.internal Ready worker 4h26m v1.22.8+9e95cb9 qiowang-10103-57fmp-worker-c-qg6gc.c.openshift-qe.internal Ready worker 4h23m v1.22.8+9e95cb9 2. Run tcpdump across all workers at once [cloud-user@preserved-qiowang must-gather-test]$ oc adm must-gather \ > --dest-dir /tmp/captures \ > --source-dir '/tmp/tcpdump/' \ > --image registry.redhat.io/openshift4/network-tools-rhel8:latest \ > --node-selector 'node-role.kubernetes.io/worker' \ > --host-network=true \ > -- \ > timeout 30 tcpdump \ > -i any \ > -w /tmp/tcpdump/\$POD_NAME-%Y-%m-%dT%H:%M:%S.pcap -W 1 -G 300 [must-gather ] OUT Using must-gather plug-in image: registry.redhat.io/openshift4/network-tools-rhel8:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a992b74f-2e32-4914-b457-fda64029028c ClusterVersion: Stable at "4.9.0-0.nightly-2022-10-10-022606" ClusterOperators: All healthy and stable [must-gather ] OUT namespace/openshift-must-gather-n86vh created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-psfgx created [must-gather ] OUT pod: must-gather-kp8bv on node: qiowang-10103-57fmp-worker-a-lmdtl.c.openshift-qe.internal for plug-in image registry.redhat.io/openshift4/network-tools-rhel8:latest created [must-gather ] OUT pod: must-gather-2kmjt on node: qiowang-10103-57fmp-worker-b-lv5k8.c.openshift-qe.internal for plug-in image registry.redhat.io/openshift4/network-tools-rhel8:latest created [must-gather ] OUT pod: must-gather-pqdrg on node: qiowang-10103-57fmp-worker-c-qg6gc.c.openshift-qe.internal for plug-in image registry.redhat.io/openshift4/network-tools-rhel8:latest created [must-gather-pqdrg] POD 2022-10-10T15:14:44.472578249Z dropped privs to tcpdump [must-gather-pqdrg] POD 2022-10-10T15:14:44.472716045Z tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes [must-gather-2kmjt] POD 2022-10-10T15:14:44.709177368Z dropped privs to tcpdump [must-gather-2kmjt] POD 2022-10-10T15:14:44.709177368Z tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes [must-gather-kp8bv] POD 2022-10-10T15:14:51.704700643Z dropped privs to tcpdump [must-gather-kp8bv] POD 2022-10-10T15:14:51.704700643Z tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes [must-gather-pqdrg] POD 2022-10-10T15:15:14.408494315Z 6583 packets captured [must-gather-pqdrg] POD 2022-10-10T15:15:14.408494315Z 6678 packets received by filter [must-gather-pqdrg] POD 2022-10-10T15:15:14.408494315Z 0 packets dropped by kernel [must-gather-2kmjt] POD 2022-10-10T15:15:14.650190061Z 7278 packets captured [must-gather-2kmjt] POD 2022-10-10T15:15:14.650190061Z 7729 packets received by filter [must-gather-2kmjt] POD 2022-10-10T15:15:14.650190061Z 0 packets dropped by kernel [must-gather-pqdrg] OUT waiting for gather to complete [must-gather-pqdrg] OUT downloading gather output [must-gather-2kmjt] OUT waiting for gather to complete [must-gather-2kmjt] OUT downloading gather output [must-gather-pqdrg] OUT receiving incremental file list [must-gather-pqdrg] OUT ./ [must-gather-pqdrg] OUT must-gather-pqdrg-2022-10-10T15:14:44.pcap [must-gather-2kmjt] OUT receiving incremental file list [must-gather-2kmjt] OUT ./ [must-gather-2kmjt] OUT must-gather-2kmjt-2022-10-10T15:14:44.pcap [must-gather-pqdrg] OUT [must-gather-pqdrg] OUT sent 46 bytes received 4,173,943 bytes 1,669,595.60 bytes/sec [must-gather-pqdrg] OUT total size is 7,893,781 speedup is 1.89 [must-gather-2kmjt] OUT [must-gather-2kmjt] OUT sent 46 bytes received 1,977,325 bytes 1,318,247.33 bytes/sec [must-gather-2kmjt] OUT total size is 4,167,949 speedup is 2.11 [must-gather-kp8bv] POD 2022-10-10T15:15:21.640402814Z 28573 packets captured [must-gather-kp8bv] POD 2022-10-10T15:15:21.640402814Z 28904 packets received by filter [must-gather-kp8bv] POD 2022-10-10T15:15:21.640402814Z 221 packets dropped by kernel [must-gather-kp8bv] OUT waiting for gather to complete [must-gather-kp8bv] OUT downloading gather output [must-gather-kp8bv] OUT receiving incremental file list [must-gather-kp8bv] OUT ./ [must-gather-kp8bv] OUT must-gather-kp8bv-2022-10-10T15:14:51.pcap [must-gather-kp8bv] OUT [must-gather-kp8bv] OUT sent 46 bytes received 22,842,125 bytes 9,136,868.40 bytes/sec [must-gather-kp8bv] OUT total size is 41,500,064 speedup is 1.82 [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-psfgx deleted [must-gather ] OUT namespace/openshift-must-gather-n86vh deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a992b74f-2e32-4914-b457-fda64029028c ClusterVersion: Stable at "4.9.0-0.nightly-2022-10-10-022606" ClusterOperators: All healthy and stable 3. Check pcap files from all 3 workers are copied to local [cloud-user@preserved-qiowang must-gather-test]$ ls -l /tmp/captures/*/* /tmp/captures/qiowang-10103-57fmp-worker-a-lmdtl-c-openshift-qe-internal/registry-redhat-io-openshift4-network-tools-rhel8-sha256-0d96329b253810cf1296ad79373867ec348da2dfe9f6d7d98e6756aeac98c5ca: total 40528 -rw-r--r--. 1 cloud-user cloud-user 41500064 Oct 10 11:15 must-gather-kp8bv-2022-10-10T15:14:51.pcap /tmp/captures/qiowang-10103-57fmp-worker-b-lv5k8-c-openshift-qe-internal/registry-redhat-io-openshift4-network-tools-rhel8-sha256-0d96329b253810cf1296ad79373867ec348da2dfe9f6d7d98e6756aeac98c5ca: total 4072 -rw-r--r--. 1 cloud-user cloud-user 4167949 Oct 10 11:15 must-gather-2kmjt-2022-10-10T15:14:44.pcap /tmp/captures/qiowang-10103-57fmp-worker-c-qg6gc-c-openshift-qe-internal/registry-redhat-io-openshift4-network-tools-rhel8-sha256-0d96329b253810cf1296ad79373867ec348da2dfe9f6d7d98e6756aeac98c5ca: total 7712 -rw-r--r--. 1 cloud-user cloud-user 7893781 Oct 10 11:15 must-gather-pqdrg-2022-10-10T15:14:44.pcap Test passed.