https://issues.redhat.com/browse/NETOBSERV-147 *Deployed NO operator and deployed goflow-kube: memodi@memodi-mac:/Users/memodi/workspaces/repos/netobserv/network-observability-operator (main *=) $ make deploy /Users/memodi/workspaces/repos/netobserv/network-observability-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases cd config/manager && /Users/memodi/workspaces/repos/netobserv/network-observability-operator/bin/kustomize edit set image controller=quay.io/netobserv/network-observability-operator:main /Users/memodi/workspaces/repos/netobserv/network-observability-operator/bin/kustomize build config/default | kubectl apply -f - namespace/network-observability created customresourcedefinition.apiextensions.k8s.io/flowcollectors.flows.netobserv.io created serviceaccount/netobserv-controller-manager created role.rbac.authorization.k8s.io/netobserv-leader-election-role created clusterrole.rbac.authorization.k8s.io/netobserv-manager-role created clusterrole.rbac.authorization.k8s.io/netobserv-metrics-reader created clusterrole.rbac.authorization.k8s.io/netobserv-proxy-role created rolebinding.rbac.authorization.k8s.io/netobserv-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/netobserv-manager-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/netobserv-proxy-rolebinding created configmap/netobserv-manager-config created service/netobserv-metrics-service created deployment.apps/netobserv-controller-manager created memodi@memodi-mac:/Users/memodi/workspaces/repos/netobserv/network-observability-operator (main *=) *Deploy goflow: $ oc apply -f config/samples/flows_v1alpha1_flowcollector.yaml flowcollector.flows.netobserv.io/cluster created *Patched CNO: $ GF_IP=`oc get svc goflow-kube -n network-observability -ojsonpath='{.spec.clusterIP}'` && echo $GF_IP 172.30.25.190 *Watched ovnkube-node restarts: ovnkube-node-kvl27 5/5 Terminating 0 5m16s ovnkube-node-kvl27 0/5 Terminating 0 5m18s ovnkube-node-kvl27 0/5 Terminating 0 5m18s ovnkube-node-kvl27 0/5 Terminating 0 5m18s ovnkube-node-dmk2j 0/5 Pending 0 0s ovnkube-node-dmk2j 0/5 Pending 0 0s ovnkube-node-dmk2j 0/5 ContainerCreating 0 0s ovnkube-node-dmk2j 4/5 Running 0 1s * one of the ovnkube-node logs: + [[ -n 172.30.25.190:2055,172.30.25.190:2055 ]] + export_network_flows_flags=' --ipfix-targets 172.30.25.190:2055,172.30.25.190:2055' + [[ -n 400 ]] + export_network_flows_flags=' --ipfix-targets 172.30.25.190:2055,172.30.25.190:2055 --ipfix-cache-max-flows 400' + [[ -n 60 ]] + export_network_flows_flags=' --ipfix-targets 172.30.25.190:2055,172.30.25.190:2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60' + [[ -n 100 ]] + export_network_flows_flags=' --ipfix-targets 172.30.25.190:2055,172.30.25.190:2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60 --ipfix-sampling 100' + gw_interface_flag= + '[' -d /sys/class/net/br-ex1 ']' + node_mgmt_port_netdev_flags= + [[ -n '' ]] + exec /usr/bin/ovnkube --init-node ci-ln-v9itd8t-72292-z87hv-master-1 --nb-address ssl:10.0.0.3:9641,ssl:10.0.0.4:9641,ssl:10.0.0.5:9641 --sb-address ssl:10.0.0.3:9642,ssl:10.0.0.4:9642,ssl:10.0.0.5:9642 --nb-client-privkey /ovn-cert/tls.key --nb-client-cert /ovn-cert/tls.crt --nb-client-cacert /ovn-ca/ca-bundle.crt --nb-cert-common-name ovn --sb-client-privkey /ovn-cert/tls.key --sb-client-cert /ovn-cert/tls.crt --sb-client-cacert /ovn-ca/ca-bundle.crt --sb-cert-common-name ovn --config-file=/run/ovnkube-config/ovnkube.conf --loglevel 4 --inactivity-probe=180000 --gateway-mode shared --gateway-interface br-ex --metrics-bind-address 127.0.0.1:29103 --ovn-metrics-bind-address 127.0.0.1:29105 --metrics-enable-pprof --ipfix-targets 172.30.25.190:2055,172.30.25.190:2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60 --ipfix-sampling 100 ... ... I0126 17:11:27.646428 197562 ovs.go:204] exec(3): /usr/bin/ovs-vsctl --timeout=15 -- --id=@ipfix create ipfix targets=["172.30.25.190:2055"] cache_active_timeout=60 cache_max_flows=400 sampling=100 -- set bridge br-int ipfix=@ipfix I0126 17:11:27.661525 197562 ovs.go:207] exec(3): stdout: "f00efcd5-25a5-463a-88cb-9a0b842ea95a\n" I0126 17:11:27.661661 197562 ovs.go:208] exec(3): stderr: "" Starts with using 2 IPFix collector IP duplicates, but in the end able to filter one of them. Confirmed all ovnkube-nodes are running properly: $ oc get pods -n openshift-ovn-kubernetes -w NAME READY STATUS RESTARTS AGE ovnkube-master-9ldd9 6/6 Running 6 (66m ago) 67m ovnkube-master-ttg4c 6/6 Running 2 (8m27s ago) 67m ovnkube-master-x6vwv 6/6 Running 6 (66m ago) 67m ovnkube-node-2ftx5 5/5 Running 0 3m50s ovnkube-node-2jdqb 5/5 Running 0 3m16s ovnkube-node-4r6wt 5/5 Running 0 3m3s ovnkube-node-5rw7v 5/5 Running 0 2m51s ovnkube-node-dmk2j 5/5 Running 0 4m2s ovnkube-node-mxccc 5/5 Running 0 3m39s *Updated patch to remove the exportNetworkFlows: $ oc patch networks.operator.openshift.io cluster --type='json' -p "$(sed -e "s/GF_IP/$GF_IP/" ~/workspaces/cluster_bot/net-cluster-patch.json)" network.operator.openshift.io/cluster patched $ oc get pods -n openshift-ovn-kubernetes -w NAME READY STATUS RESTARTS AGE ovnkube-master-9ldd9 6/6 Running 6 (76m ago) 77m ovnkube-master-ttg4c 6/6 Running 2 (18m ago) 77m ovnkube-master-x6vwv 6/6 Running 6 (76m ago) 77m ovnkube-node-2jdqb 5/5 Running 0 13m ovnkube-node-5rw7v 5/5 Running 0 12m ovnkube-node-64pzg 5/5 Running 0 43s ovnkube-node-9dvgr 4/5 Running 0 3s ovnkube-node-f8h6h 5/5 Running 0 31s ovnkube-node-wr9px 5/5 Running 0 19s * logs from ovnkube-node: export_network_flows_flags=' --ipfix-targets 172.30.25.190:2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60 --ipfix-sampling 100' + gw_interface_flag= + '[' -d /sys/class/net/br-ex1 ']' + node_mgmt_port_netdev_flags= + [[ -n '' ]] + exec /usr/bin/ovnkube --init-node ci-ln-v9itd8t-72292-z87hv-master-0 --nb-address ssl:10.0.0.3:9641,ssl:10.0.0.4:9641,ssl:10.0.0.5:9641 --sb-address ssl:10.0.0.3: 9642,ssl:10.0.0.4:9642,ssl:10.0.0.5:9642 --nb-client-privkey /ovn-cert/tls.key --nb-client-cert /ovn-cert/tls.crt --nb-client-cacert /ovn-ca/ca-bundle.crt --nb-cert -common-name ovn --sb-client-privkey /ovn-cert/tls.key --sb-client-cert /ovn-cert/tls.crt --sb-client-cacert /ovn-ca/ca-bundle.crt --sb-cert-common-name ovn --confi g-file=/run/ovnkube-config/ovnkube.conf --loglevel 4 --inactivity-probe=180000 --gateway-mode shared --gateway-interface br-ex --metrics-bind-address 127.0.0.1:2910 3 --ovn-metrics-bind-address 127.0.0.1:29105 --metrics-enable-pprof --ipfix-targets 172.30.25.190:2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60 - -ipfix-sampling 100 * go flow is receiving flows fine: $ oc logs pod/goflow-kube-6d7469478d-sp4bc | tail time="2022-01-26T17:28:02Z" level=warning msg="Failed to get Service [ip=10.128.2.2]" module=goflow-kube {"BiFlowDirection":0,"Bytes":24900,"CustomBytes1":null,"CustomBytes2":null,"CustomInteger1":0,"CustomInteger2":0,"DstAS":0,"DstAddr":"10.128.2.2","DstMac":"ae:a6:1e:07:c6:56","DstNet":0,"DstPort":51714,"DstVlan":0,"EgressVrfID":0,"Etype":2048,"FlowDirection":1,"ForwardingStatus":0,"FragmentId":0,"FragmentOffset":0,"HasMPLS":false,"IPTTL":0,"IPTos":0,"IPv6FlowLabel":0,"IcmpCode":0,"IcmpType":0,"InIf":15,"IngressVrfID":0,"MPLS1Label":0,"MPLS1TTL":0,"MPLS2Label":0,"MPLS2TTL":0,"MPLS3Label":0,"MPLS3TTL":0,"MPLSCount":0,"MPLSLastLabel":0,"MPLSLastTTL":0,"NextHop":null,"NextHopAS":0,"OutIf":10,"Packets":100,"Proto":6,"SamplerAddress":"ZEAABg==","SamplingRate":0,"SequenceNum":191,"SrcAS":0,"SrcAddr":"10.128.2.6","SrcHostIP":"10.0.128.4","SrcMac":"0a:58:0a:80:02:06","SrcNamespace":"openshift-monitoring","SrcNet":0,"SrcPod":"thanos-querier-5644dffbdb-lgxk8","SrcPort":9091,"SrcVlan":0,"SrcWorkload":"thanos-querier","SrcWorkloadKind":"Deployment","TCPFlags":0,"TimeFlowEnd":1643218023,"TimeFlowStart":1643218023,"TimeReceived":1643218082,"Type":4,"VlanId":0} time="2022-01-26T17:28:03Z" level=info component=client error="Post \"http://loki:3100/loki/api/v1/push\": dial tcp: lookup loki on 172.30.0.10:53: no such host" fields.level=warn fields.msg="error sending batch, will retry" host="loki:3100" module=export/loki status=-1 {"BiFlowDirection":0,"Bytes":5200,"CustomBytes1":null,"CustomBytes2":null,"CustomInteger1":0,"CustomInteger2":0,"DstAS":0,"DstAddr":"10.131.0.21","DstHostIP":"10.0.128.3","DstMac":"0a:58:0a:80:02:01","DstNamespace":"openshift-monitoring","DstNet":0,"DstPod":"prometheus-k8s-0","DstPort":10901,"DstVlan":0,"DstWorkload":"prometheus-k8s","DstWorkloadKind":"StatefulSet","EgressVrfID":0,"Etype":2048,"FlowDirection":0,"ForwardingStatus":0,"FragmentId":0,"FragmentOffset":0,"HasMPLS":false,"IPTTL":0,"IPTos":0,"IPv6FlowLabel":0,"IcmpCode":0,"IcmpType":0,"InIf":15,"IngressVrfID":0,"MPLS1Label":0,"MPLS1TTL":0,"MPLS2Label":0,"MPLS2TTL":0,"MPLS3Label":0,"MPLS3TTL":0,"MPLSCount":0,"MPLSLastLabel":0,"MPLSLastTTL":0,"NextHop":null,"NextHopAS":0,"OutIf":0,"Packets":100,"Proto":6,"SamplerAddress":"ZEAABg==","SamplingRate":0,"SequenceNum":192,"SrcAS":0,"SrcAddr":"10.128.2.6","SrcHostIP":"10.0.128.4","SrcMac":"0a:58:0a:80:02:06","SrcNamespace":"openshift-monitoring","SrcNet":0,"SrcPod":"thanos-querier-5644dffbdb-lgxk8","SrcPort":45736,"SrcVlan":0,"SrcWorkload":"thanos-querier","SrcWorkloadKind":"Deployment","TCPFlags":0,"TimeFlowEnd":1643218024,"TimeFlowStart":1643218024,"TimeReceived":1643218083,"Type":4,"VlanId":0} *trying go flow as DaemonSet $ oc get pods NAME READY STATUS RESTARTS AGE goflow-kube-7glfn 1/1 Running 0 72s goflow-kube-82vjc 1/1 Running 0 72s goflow-kube-bgs7k 1/1 Running 0 72s goflow-kube-g82c9 1/1 Running 0 72s goflow-kube-s2jbm 1/1 Running 0 72s goflow-kube-trjgv 1/1 Running 0 72s netobserv-controller-manager-5759bc6964-tc5ft 2/2 Running 0 27m network-observability-plugin-85dc9f5685-vwm4v 1/1 Running 0 27m $ oc logs pod/ovnkube-node-xmgmz ovnkube-node -n openshift-ovn-kubernetes | egrep -i -A 2 ipfix + export_network_flows_flags=' --ipfix-targets :2055' + [[ -n 400 ]] + export_network_flows_flags=' --ipfix-targets :2055 --ipfix-cache-max-flows 400' + [[ -n 60 ]] + export_network_flows_flags=' --ipfix-targets :2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60' + [[ -n 100 ]] + export_network_flows_flags=' --ipfix-targets :2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60 --ipfix-sampling 100' + gw_interface_flag= + '[' -d /sys/class/net/br-ex1 ']' -- + exec /usr/bin/ovnkube --init-node ci-ln-v9itd8t-72292-z87hv-master-2 --nb-address ssl:10.0.0.3:9641,ssl:10.0.0.4:9641,ssl:10.0.0.5:9641 --sb-address ssl:10.0.0.3:9642,ssl:10.0.0.4:9642,ssl:10.0.0.5:9642 --nb-client-privkey /ovn-cert/tls.key --nb-client-cert /ovn-cert/tls.crt --nb-client-cacert /ovn-ca/ca-bundle.crt --nb-cert-common-name ovn --sb-client-privkey /ovn-cert/tls.key --sb-client-cert /ovn-cert/tls.crt --sb-client-cacert /ovn-ca/ca-bundle.crt --sb-cert-common-name ovn --config-file=/run/ovnkube-config/ovnkube.conf --loglevel 4 --inactivity-probe=180000 --gateway-mode shared --gateway-interface br-ex --metrics-bind-address 127.0.0.1:29103 --ovn-metrics-bind-address 127.0.0.1:29105 --metrics-enable-pprof --ipfix-targets :2055 --ipfix-cache-max-flows 400 --ipfix-cache-active-timeout 60 --ipfix-sampling 100 I0126 17:31:23.892139 258163 ovs.go:93] Maximum command line arguments set to: 191102 I0126 17:31:23.895510 258163 config.go:1674] Parsed config file /run/ovnkube-config/ovnkube.conf -- I0126 17:31:23.895549 258163 config.go:1675] Parsed config: {Default:{MTU:1360 RoutableMTU:0 ConntrackZone:64000 EncapType:geneve EncapIP: EncapPort:6081 InactivityProbe:100000 OpenFlowProbe:180 MonitorAll:true LFlowCacheEnable:true LFlowCacheLimit:0 LFlowCacheLimitKb:1048576 RawClusterSubnets:10.128.0.0/14/23 ClusterSubnets:[]} Logging:{File: CNIFile: Level:4 LogFileMaxSize:100 LogFileMaxBackups:5 LogFileMaxAge:5 ACLLoggingRateLimit:20} Monitoring:{RawNetFlowTargets: RawSFlowTargets: RawIPFIXTargets: NetFlowTargets:[] SFlowTargets:[] IPFIXTargets:[]} IPFIX:{Sampling:400 CacheActiveTimeout:60 CacheMaxFlows:0} CNI:{ConfDir:/etc/cni/net.d Plugin:ovn-k8s-cni-overlay} OVNKubernetesFeature:{EnableEgressIP:true EnableEgressFirewall:true} Kubernetes:{Kubeconfig: CACert: CAData:[] APIServer:https://api-int.ci-ln-v9itd8t-72292.origin-ci-int-gce.dev.rhcloud.com:6443 Token: CompatServiceCIDR: RawServiceCIDRs:172.30.0.0/16 ServiceCIDRs:[] OVNConfigNamespace:openshift-ovn-kubernetes MetricsBindAddress: OVNMetricsBindAddress: MetricsEnablePprof:false OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:nil HostNetworkNamespace:openshift-host-network PlatformType:GCP} OvnNorth:{Address: PrivKey: Cert: CACert: CertCommonName: Scheme: ElectionTimer:0 northbound:false exec:} OvnSouth:{Address: PrivKey: Cert: CACert: CertCommonName: Scheme: ElectionTimer:0 northbound:false exec:} Gateway:{Mode:shared Interface: EgressGWInterface: NextHop: VLANID:0 NodeportEnable:true DisableSNATMultipleGWs:false V4JoinSubnet:100.64.0.0/16 V6JoinSubnet:fd98::/64 DisablePacketMTUCheck:false RouterSubnet:} MasterHA:{ElectionLeaseDuration:60 ElectionRenewDeadline:30 ElectionRetryPeriod:20} HybridOverlay:{Enabled:false RawClusterSubnets: ClusterSubnets:[] VXLANPort:4789} OvnKubeNode:{Mode:full MgmtPortNetdev: DisableOVNIfaceIdVer:false}} I0126 17:31:23.899484 258163 node.go:330] OVN Kube Node initialization, Mode: full I0126 17:31:23.899980 258163 reflector.go:219] Starting reflector *v1.Endpoints (0s) from k8s.io/client-go/informers/factory.go:134 -- I0126 17:31:24.055041 258163 ovs.go:204] exec(2): /usr/bin/ovs-vsctl --timeout=15 -- clear bridge br-int netflow -- clear bridge br-int sflow -- clear bridge br-int ipfix I0126 17:31:24.068617 258163 ovs.go:207] exec(2): stdout: "" I0126 17:31:24.068775 258163 ovs.go:208] exec(2): stderr: "" -- I0126 17:31:24.068876 258163 ovs.go:204] exec(3): /usr/bin/ovs-vsctl --timeout=15 -- --id=@ipfix create ipfix targets=["10.0.0.4:2055"] cache_active_timeout=60 cache_max_flows=400 sampling=100 -- set bridge br-int ipfix=@ipfix I0126 17:31:24.086169 258163 ovs.go:207] exec(3): stdout: "744f2efa-492b-41bb-8a09-8ac09b7b2e4e\n" I0126 17:31:24.086284 258163 ovs.go:208] exec(3): stderr: "" $ oc logs pod/goflow-kube-trjgv | tail {"BiFlowDirection":0,"Bytes":10700,"CustomBytes1":null,"CustomBytes2":null,"CustomInteger1":0,"CustomInteger2":0,"DstAS":0,"DstAddr":"10.129.0.2","DstMac":"6e:dd:0d:e5:c3:8e","DstNet":0,"DstPort":49788,"DstVlan":0,"EgressVrfID":0,"Etype":2048,"FlowDirection":1,"ForwardingStatus":0,"FragmentId":0,"FragmentOffset":0,"HasMPLS":false,"IPTTL":0,"IPTos":0,"IPv6FlowLabel":0,"IcmpCode":0,"IcmpType":0,"InIf":47,"IngressVrfID":0,"MPLS1Label":0,"MPLS1TTL":0,"MPLS2Label":0,"MPLS2TTL":0,"MPLS3Label":0,"MPLS3TTL":0,"MPLSCount":0,"MPLSLastLabel":0,"MPLSLastTTL":0,"NextHop":null,"NextHopAS":0,"OutIf":9,"Packets":100,"Proto":6,"SamplerAddress":"CoAAAg==","SamplingRate":0,"SequenceNum":2,"SrcAS":0,"SrcAddr":"10.128.0.35","SrcHostIP":"10.0.0.3","SrcMac":"0a:58:0a:81:00:01","SrcNamespace":"openshift-apiserver","SrcNet":0,"SrcPod":"apiserver-74f94f7f7d-258n4","SrcPort":8443,"SrcVlan":0,"SrcWorkload":"apiserver","SrcWorkloadKind":"Deployment","TCPFlags":0,"TimeFlowEnd":1643218309,"TimeFlowStart":1643218309,"TimeReceived":1643218368,"Type":4,"VlanId":0} time="2022-01-26T17:32:50Z" level=info component=client error="Post \"http://loki:3100/loki/api/v1/push\": dial tcp: lookup loki on 172.30.0.10:53: no such host" fields.level=warn fields.msg="error sending batch, will retry" host="loki:3100" module=export/loki status=-1 time="2022-01-26T17:32:50Z" level=warning msg="Failed to get Service [ip=10.128.0.2]" module=goflow-kube {"BiFlowDirection":0,"Bytes":5200,"CustomBytes1":null,"CustomBytes2":null,"CustomInteger1":0,"CustomInteger2":0,"DstAS":0,"DstAddr":"10.128.0.25","DstHostIP":"10.0.0.3","DstMac":"0a:58:0a:80:00:19","DstNamespace":"openshift-oauth-apiserver","DstNet":0,"DstPod":"apiserver-7459d9bdfc-r4jvr","DstPort":38962,"DstVlan":0,"DstWorkload":"apiserver","DstWorkloadKind":"Deployment","EgressVrfID":0,"Etype":2048,"FlowDirection":0,"ForwardingStatus":0,"FragmentId":0,"FragmentOffset":0,"HasMPLS":false,"IPTTL":0,"IPTos":0,"IPv6FlowLabel":0,"IcmpCode":0,"IcmpType":0,"InIf":10,"IngressVrfID":0,"MPLS1Label":0,"MPLS1TTL":0,"MPLS2Label":0,"MPLS2TTL":0,"MPLS3Label":0,"MPLS3TTL":0,"MPLSCount":0,"MPLSLastLabel":0,"MPLSLastTTL":0,"NextHop":null,"NextHopAS":0,"OutIf":0,"Packets":100,"Proto":6,"SamplerAddress":"CoAAAg==","SamplingRate":0,"SequenceNum":3,"SrcAS":0,"SrcAddr":"10.128.0.2","SrcMac":"f6:a5:98:24:b8:64","SrcNet":0,"SrcPort":2379,"SrcVlan":0,"TCPFlags":0,"TimeFlowEnd":1643218310,"TimeFlowStart":1643218310,"TimeReceived":1643218370,"Type":4,"VlanId":0}