https://issues.redhat.com/browse/SDN-1697 Single stack IPv4 ovnkube-trace works fine: $ oc exec -ti ovn-trace -- ovnkube-trace -dst-namespace ovn-tests -dst ovn-trace -src-namespace ovn-tests -src ovn-trace-two -tcp -loglevel 10 I1104 15:32:16.323828 10 ovs.go:98] Maximum command line arguments set to: 191102 I1104 15:32:16.323988 10 ovnkube-trace.go:463] Log level set to: 10 I1104 15:32:16.387178 10 ovnkube-trace.go:545] OVN Kubernetes namespace is openshift-ovn-kubernetes I1104 15:32:16.397989 10 ovnkube-trace.go:557] Nodes: I1104 15:32:16.398021 10 ovnkube-trace.go:562] Name: ci-ln-tp3hvw2-72292-t8rf7-master-0 is a master I1104 15:32:16.398059 10 ovnkube-trace.go:564] Address Type: InternalIP - Address: 10.0.0.5 I1104 15:32:16.398128 10 ovnkube-trace.go:564] Address Type: InternalDNS - Address: ci-ln-tp3hvw2-72292-t8rf7-master-0.c.openshift-gce-devel-ci.internal I1104 15:32:16.398182 10 ovnkube-trace.go:564] Address Type: Hostname - Address: ci-ln-tp3hvw2-72292-t8rf7-master-0.c.openshift-gce-devel-ci.internal I1104 15:32:16.398208 10 ovnkube-trace.go:562] Name: ci-ln-tp3hvw2-72292-t8rf7-master-1 is a master I1104 15:32:16.398259 10 ovnkube-trace.go:564] Address Type: InternalIP - Address: 10.0.0.3 I1104 15:32:16.398315 10 ovnkube-trace.go:564] Address Type: InternalDNS - Address: ci-ln-tp3hvw2-72292-t8rf7-master-1.c.openshift-gce-devel-ci.internal I1104 15:32:16.398362 10 ovnkube-trace.go:564] Address Type: Hostname - Address: ci-ln-tp3hvw2-72292-t8rf7-master-1.c.openshift-gce-devel-ci.internal I1104 15:32:16.398380 10 ovnkube-trace.go:562] Name: ci-ln-tp3hvw2-72292-t8rf7-master-2 is a master I1104 15:32:16.398449 10 ovnkube-trace.go:564] Address Type: InternalIP - Address: 10.0.0.4 I1104 15:32:16.398501 10 ovnkube-trace.go:564] Address Type: InternalDNS - Address: ci-ln-tp3hvw2-72292-t8rf7-master-2.c.openshift-gce-devel-ci.internal I1104 15:32:16.398547 10 ovnkube-trace.go:564] Address Type: Hostname - Address: ci-ln-tp3hvw2-72292-t8rf7-master-2.c.openshift-gce-devel-ci.internal I1104 15:32:16.398601 10 ovnkube-trace.go:571] Name: ci-ln-tp3hvw2-72292-t8rf7-worker-a-w7vjp is a worker I1104 15:32:16.398650 10 ovnkube-trace.go:573] Address Type: InternalIP - Address: 10.0.128.3 I1104 15:32:16.398699 10 ovnkube-trace.go:573] Address Type: InternalDNS - Address: ci-ln-tp3hvw2-72292-t8rf7-worker-a-w7vjp.c.openshift-gce-devel-ci.internal I1104 15:32:16.398753 10 ovnkube-trace.go:573] Address Type: Hostname - Address: ci-ln-tp3hvw2-72292-t8rf7-worker-a-w7vjp.c.openshift-gce-devel-ci.internal I1104 15:32:16.398809 10 ovnkube-trace.go:571] Name: ci-ln-tp3hvw2-72292-t8rf7-worker-b-wmt8r is a worker I1104 15:32:16.398859 10 ovnkube-trace.go:573] Address Type: InternalIP - Address: 10.0.128.4 I1104 15:32:16.398910 10 ovnkube-trace.go:573] Address Type: InternalDNS - Address: ci-ln-tp3hvw2-72292-t8rf7-worker-b-wmt8r.c.openshift-gce-devel-ci.internal I1104 15:32:16.398957 10 ovnkube-trace.go:573] Address Type: Hostname - Address: ci-ln-tp3hvw2-72292-t8rf7-worker-b-wmt8r.c.openshift-gce-devel-ci.internal I1104 15:32:16.399004 10 ovnkube-trace.go:571] Name: ci-ln-tp3hvw2-72292-t8rf7-worker-c-4hpfp is a worker I1104 15:32:16.399052 10 ovnkube-trace.go:573] Address Type: InternalIP - Address: 10.0.128.2 I1104 15:32:16.399107 10 ovnkube-trace.go:573] Address Type: InternalDNS - Address: ci-ln-tp3hvw2-72292-t8rf7-worker-c-4hpfp.c.openshift-gce-devel-ci.internal I1104 15:32:16.399152 10 ovnkube-trace.go:573] Address Type: Hostname - Address: ci-ln-tp3hvw2-72292-t8rf7-worker-c-4hpfp.c.openshift-gce-devel-ci.internal I1104 15:32:16.399167 10 ovnkube-trace.go:597] Master name: ci-ln-tp3hvw2-72292-t8rf7-master-0 has IP 10.0.0.5 I1104 15:32:16.399182 10 ovnkube-trace.go:597] Master name: ci-ln-tp3hvw2-72292-t8rf7-master-1 has IP 10.0.0.3 I1104 15:32:16.399194 10 ovnkube-trace.go:597] Master name: ci-ln-tp3hvw2-72292-t8rf7-master-2 has IP 10.0.0.4 I1104 15:32:16.399214 10 ovnkube-trace.go:611] The nbcmd is -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db ssl:10.0.0.5:9641,ssl:10.0.0.3:9641,ssl:10.0.0.4:9641 I1104 15:32:16.399236 10 ovnkube-trace.go:629] The sbcmd is -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db ssl:10.0.0.5:9642,ssl:10.0.0.3:9642,ssl:10.0.0.4:9642 I1104 15:32:16.415321 10 ovnkube-trace.go:243] ==>Got pod ovn-trace-two which is running on node ci-ln-tp3hvw2-72292-t8rf7-worker-a-w7vjp I1104 15:32:16.415508 10 ovnkube-trace.go:267] Reading interface index from /sys/class/net/... I1104 15:32:16.415552 10 ovnkube-trace.go:270] The command is cat /sys/class/net/eth0/iflink I1104 15:32:16.518972 10 ovnkube-trace.go:284] Using 'cat /sys/class/net/eth0/iflink' - linkIndex is 37 I1104 15:32:16.519005 10 ovnkube-trace.go:286] Using interface name of eth0 with MAC of 0a:58:0a:80:02:20 I1104 15:32:16.543922 10 ovnkube-trace.go:310] ==> pod ovnkube-node-q4sfq is running on node ci-ln-tp3hvw2-72292-t8rf7-worker-a-w7vjp I1104 15:32:16.543957 10 ovnkube-trace.go:328] Command is: ovn-nbctl -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db ssl:10.0.0.5:9641,ssl:10.0.0.3:9641,ssl:10.0.0.4:9641 lsp-get-addresses stor-ci-ln-tp3hvw2-72292-t8rf7-worker-a-w7vjp I1104 15:32:16.686735 10 ovnkube-trace.go:351] Command is: ip -j addr show ... ... Found bugs on for dual stack and IPv6 https://bugzilla.redhat.com/show_bug.cgi?id=2019979 https://bugzilla.redhat.com/show_bug.cgi?id=2020311 Discussions: https://coreos.slack.com/archives/GLDDW02SJ/p1635877790096500 https://coreos.slack.com/archives/CCQEHQSPQ/p1635962257422700 Re-veifying after above bugs are on ON_QA: $ oc adm release info --image-for=ovn-kubernetes quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08511567eb709ef7144e088321906d3da81697fb903c3a7c6f75f2ce1effa016 $ egrep image ovnkube-trace* ovnkube-trace.yaml: image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08511567eb709ef7144e088321906d3da81697fb903c3a7c6f75f2ce1effa016 ovnkube-trace2.yaml: image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08511567eb709ef7144e088321906d3da81697fb903c3a7c6f75f2ce1effa016 $ oc apply -f ovnkube-trace.yaml clusterrole.rbac.authorization.k8s.io/priv created clusterrolebinding.rbac.authorization.k8s.io/priv created pod/ovn-trace created memodi@memodi-mac:/Users/memodi/workspaces/cluster_bot $ oc apply -f ovnkube-trace2.yaml clusterrole.rbac.authorization.k8s.io/priv unchanged clusterrolebinding.rbac.authorization.k8s.io/priv unchanged pod/ovn-trace-two created memodi@memodi-mac:/Users/memodi/workspaces/cluster_bot Every 2.0s: oc get pods -o wide memodi-mac: Tue Nov 30 16:48:08 2021 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ovn-trace 1/1 Running 0 66s 10.128.2.72 worker-01.memodi-dualsec5.qe.devcluster.openshift.com ovn-trace-two 1/1 Running 0 62s 10.128.2.73 worker-01.memodi-dualsec5.qe.devcluster.openshift.com -- Verifying on dual stack cluster: $ oc adm release info --image-for=ovn-kubernetes quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70876c20a885c88864d9bfb4abc8bb464e9f716b721a0fabf246c34f0a9283b2 Replace above image in below manifest file: $ egrep image ~/workspaces/cluster_bot/ovnkube-trace*.yaml /Users/memodi/workspaces/cluster_bot/ovnkube-trace.yaml: image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70876c20a885c88864d9bfb4abc8bb464e9f716b721a0fabf246c34f0a9283b2 /Users/memodi/workspaces/cluster_bot/ovnkube-trace2.yaml: image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70876c20a885c88864d9bfb4abc8bb464e9f716b721a0fabf246c34f0a9283b2 $ oc new-project ovn-tests Now using project "ovn-tests" on server "https://api.memodi-0108-ds.qe.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname memodi@memodi-mac:/Users/memodi/workspaces/repos/flexy-templates/functionality-testing (master=) $ oc apply -f ~/workspaces/cluster_bot/ovnkube-trace.yaml clusterrole.rbac.authorization.k8s.io/priv created clusterrolebinding.rbac.authorization.k8s.io/priv created pod/ovn-trace created memodi@memodi-mac:/Users/memodi/workspaces/repos/flexy-templates/functionality-testing (master=) $ oc apply -f ~/workspaces/cluster_bot/ovnkube-trace2.yaml clusterrole.rbac.authorization.k8s.io/priv unchanged clusterrolebinding.rbac.authorization.k8s.io/priv unchanged pod/ovn-trace-two created $ oc exec -ti ovn-trace -- ovnkube-trace -dst-namespace ovn-tests -dst ovn-trace -src-namespace ovn-tests -src ovn-trace-two -tcp I0107 23:02:02.912266 68 ovs.go:93] Maximum command line arguments set to: 191102 I0107 23:02:02.912410 68 ovnkube-trace.go:517] Log level set to: 0 ovn-trace indicates success from ovn-trace-two to ovn-trace - matched on output to "ovn-tests_ovn-trace" I0107 23:02:04.907788 68 ovnkube-trace.go:779] ovn-trace indicates success from ovn-trace-two to ovn-trace - matched on output to "ovn-tests_ovn-trace" ovn-trace indicates success from ovn-trace to ovn-trace-two - matched on output to "ovn-tests_ovn-trace-two" I0107 23:02:05.405550 68 ovnkube-trace.go:821] ovn-trace indicates success from ovn-trace to ovn-trace-two - matched on output to "ovn-tests_ovn-trace-two" ovs-appctl ofproto/trace indicates success from ovn-trace-two to ovn-trace - matched on -> output to kernel tunnel I0107 23:02:05.528876 68 ovnkube-trace.go:865] ovs-appctl ofproto/trace indicates success from ovn-trace-two to ovn-trace - matched on -> output to kernel tunnel ovs-appctl ofproto/trace indicates success from ovn-trace to ovn-trace-two - matched on -> output to kernel tunnel I0107 23:02:05.668470 68 ovnkube-trace.go:909] ovs-appctl ofproto/trace indicates success from ovn-trace to ovn-trace-two - matched on -> output to kernel tunnel ovn-trace command Completed normally memodi@memodi-mac:/Users/memodi/workspaces/repos/flexy-templates/functionality-testing (master=) Running by increasing the log level: $ oc exec -ti ovn-trace -- ovnkube-trace -dst-namespace ovn-tests -dst ovn-trace -src-namespace ovn-tests -src ovn-trace-two -tcp -loglevel 5 I0107 23:02:41.465519 113 ovs.go:93] Maximum command line arguments set to: 191102 I0107 23:02:41.465634 113 ovnkube-trace.go:517] Log level set to: 5 I0107 23:02:41.465765 113 merged_client_builder.go:121] Using in-cluster configuration I0107 23:02:41.511845 113 ovnkube-trace.go:599] OVN Kubernetes namespace is openshift-ovn-kubernetes I0107 23:02:41.516764 113 ovnkube-trace.go:611] Nodes: I0107 23:02:41.516796 113 ovnkube-trace.go:616] Name: master-00.memodi-0108-ds.qe.devcluster.openshift.com is a master I0107 23:02:41.516823 113 ovnkube-trace.go:618] Address Type: InternalIP - Address: 145.40.65.93 I0107 23:02:41.516852 113 ovnkube-trace.go:618] Address Type: InternalIP - Address: 2604:1380:0:de00::33 I0107 23:02:41.516876 113 ovnkube-trace.go:618] Address Type: Hostname - Address: master-00.memodi-0108-ds.qe.devcluster.openshift.com I0107 23:02:41.516919 113 ovnkube-trace.go:616] Name: master-01.memodi-0108-ds.qe.devcluster.openshift.com is a master I0107 23:02:41.516945 113 ovnkube-trace.go:618] Address Type: InternalIP - Address: 145.40.65.121 I0107 23:02:41.516970 113 ovnkube-trace.go:618] Address Type: InternalIP - Address: 2604:1380:0:de00::37 I0107 23:02:41.516995 113 ovnkube-trace.go:618] Address Type: Hostname - Address: master-01.memodi-0108-ds.qe.devcluster.openshift.com I0107 23:02:41.517099 113 ovnkube-trace.go:616] Name: master-02.memodi-0108-ds.qe.devcluster.openshift.com is a master I0107 23:02:41.517139 113 ovnkube-trace.go:618] Address Type: InternalIP - Address: 147.75.66.63 I0107 23:02:41.517173 113 ovnkube-trace.go:618] Address Type: InternalIP - Address: 2604:1380:0:de00::2b I0107 23:02:41.517199 113 ovnkube-trace.go:618] Address Type: Hostname - Address: master-02.memodi-0108-ds.qe.devcluster.openshift.com I0107 23:02:41.517231 113 ovnkube-trace.go:625] Name: worker-00.memodi-0108-ds.qe.devcluster.openshift.com is a worker I0107 23:02:41.517278 113 ovnkube-trace.go:627] Address Type: InternalIP - Address: 145.40.65.109 I0107 23:02:41.517303 113 ovnkube-trace.go:627] Address Type: InternalIP - Address: 2604:1380:0:de00::35 I0107 23:02:41.517352 113 ovnkube-trace.go:627] Address Type: Hostname - Address: worker-00.memodi-0108-ds.qe.devcluster.openshift.com I0107 23:02:41.517400 113 ovnkube-trace.go:625] Name: worker-01.memodi-0108-ds.qe.devcluster.openshift.com is a worker I0107 23:02:41.517426 113 ovnkube-trace.go:627] Address Type: InternalIP - Address: 145.40.65.79 I0107 23:02:41.517450 113 ovnkube-trace.go:627] Address Type: InternalIP - Address: 2604:1380:0:de00::2d I0107 23:02:41.517474 113 ovnkube-trace.go:627] Address Type: Hostname - Address: worker-01.memodi-0108-ds.qe.devcluster.openshift.com I0107 23:02:41.536693 113 ovnkube-trace.go:451] Found pod 'ovnkube-master-srwmx' with container 'ovnkube-master' I0107 23:02:41.701213 113 ovnkube-trace.go:475] Nb address for OVN database communication is ssl:145.40.65.121:9641,ssl:145.40.65.93:9641,ssl:147.75.66.63:9641 I0107 23:02:41.701240 113 ovnkube-trace.go:476] Sb address for OVN database communication is ssl:145.40.65.121:9642,ssl:145.40.65.93:9642,ssl:147.75.66.63:9642 I0107 23:02:41.701249 113 ovnkube-trace.go:477] Protocol for OVN database communication is ssl I0107 23:02:41.701265 113 ovnkube-trace.go:655] The nbcmd is -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db ssl:145.40.65.121:9641,ssl:145.40.65.93:9641,ssl:147.75.66.63:9641 I0107 23:02:41.701282 113 ovnkube-trace.go:658] The sbcmd is -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db ssl:145.40.65.121:9642,ssl:145.40.65.93:9642,ssl:147.75.66.63:9642 I0107 23:02:41.712084 113 ovnkube-trace.go:243] ==>Got pod ovn-trace-two which is running on node worker-00.memodi-0108-ds.qe.devcluster.openshift.com I0107 23:02:41.712199 113 ovnkube-trace.go:267] Reading interface index from /sys/class/net/... I0107 23:02:41.712212 113 ovnkube-trace.go:270] The command is cat /sys/class/net/eth0/iflink I0107 23:02:41.839964 113 ovnkube-trace.go:284] Using 'cat /sys/class/net/eth0/iflink' - linkIndex is 37 I0107 23:02:41.839992 113 ovnkube-trace.go:286] Using interface name of eth0 with MAC of 0a:58:0a:80:02:1b I0107 23:02:41.859537 113 ovnkube-trace.go:310] ==> pod ovnkube-node-b685q is running on node worker-00.memodi-0108-ds.qe.devcluster.openshift.com I0107 23:02:41.859567 113 ovnkube-trace.go:327] Command is: ovn-nbctl -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db ssl:145.40.65.121:9641,ssl:145.40.65.93:9641,ssl:147.75.66.63:9641 --bare --no-heading --column=mac list logical-router-port rtos-worker-00.memodi-0108-ds.qe.devcluster.openshift.com ... ... ovn-trace command Completed normally Notably, log line: I0107 23:02:41.859567 113 ovnkube-trace.go:327] Command is: ovn-nbctl -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt --db ssl:145.40.65.121:9641,ssl:145.40.65.93:9641,ssl:147.75.66.63:9641 --bare --no-heading --column=mac list logical-router-port rtos-worker-00.memodi-0108-ds.qe.devcluster.openshift.com Shows which IP interface it uses. * this was also verified on single-stack IPv6 cluster during pre-merge testing.