-
Bug
-
Resolution: Done
-
Major
-
None
-
4.18
-
Quality / Stability / Reliability
-
False
-
-
None
-
Critical
-
Yes
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
EgressIP snat rules missed or duplicated after restarting ovn pod for default network
Version-Release number of selected component (if applicable):
Pre-merge testing for 'build openshift/api#2127,openshift/ovn-kubernetes#2422' on AWS
% oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-0.ci.test-2025-01-23-014246-ci-ln-sckty2b-latest True False 6h13m Cluster version is 4.18.0-0.ci.test-2025-01-23-014246-ci-ln-sckty2b-latest
How reproducible:
Steps to Reproduce:
We have one auto case OCP-47021 frequently failed basically test is after restarting ovn pod which located same node as egress node, either duplicating snat rules left or egressIP was not applied to egress node.
I ran it for this pre-merge testing 4 times, and got 4 times failed result, below is the recorded result. Compared the testing result for 4.17.0-0.nightly-2025-01-21-205102 (AWS), the same case 4 times run, 4 times passed. I think it might be a regression for 4.18. I know there is a similar bug before https://issues.redhat.com/browse/OCPBUGS-16217 which is originally for Azure tracking, this is might different.
Considering the comparing testing result to 4.17 and egressIP is a hot feature that customer broadly used, so I raised a new bug for DEV to evaluate if needs to be fixed in 4.18.
1. First run, lr-policy-list for egressIP lost, this is probably same as below 3rd or 4th run
I0123 14:09:18.673247 46561 client.go:835] Running 'oc --kubeconfig=/tmp/kubeconfig rsh -n openshift-ovn-kubernetes ovnkube-node-s9rrc bash -c ovn-nbctl lr-policy-list ovn_cluster_router | grep -v inport' I0123 14:09:22.245998 46561 cloud_egressip_ovn.go:2968] Defaulted container "ovn-controller" out of: ovn-controller, ovn-acl-logging, kube-rbac-proxy-node, kube-rbac-proxy-ovn-metrics, northd, nbdb, sbdb, ovnkube-controller, kubecfg-setup (init) Routing Policies 102 (ip4.src == $a8519615025667110816 || ip4.src == $a13607449821398607916) && ip4.dst == $a712973235162149816 allow pkt_mark=1008 102 ip4.src == 10.128.0.0/14 && ip4.dst == 10.128.0.0/14 allow 102 ip4.src == 10.128.0.0/14 && ip4.dst == 100.64.0.0/16 allow 102 pkt.mark == 42 allow
2. Second Run, snat rules duplicated
I0123 14:18:05.489056 47011 client.go:835] Running 'oc --kubeconfig=/tmp/kubeconfig rsh -n openshift-ovn-kubernetes ovnkube-node-2xdkn bash -c ovn-nbctl lr-policy-list ovn_cluster_router | grep -v inport' I0123 14:18:11.074792 47011 cloud_egressip_ovn.go:2968] Defaulted container "ovn-controller" out of: ovn-controller, ovn-acl-logging, kube-rbac-proxy-node, kube-rbac-proxy-ovn-metrics, northd, nbdb, sbdb, ovnkube-controller, kubecfg-setup (init) Routing Policies 102 (ip4.src == $a8519615025667110816 || ip4.src == $a13607449821398607916) && ip4.dst == $a712973235162149816 allow pkt_mark=1008 102 ip4.src == 10.128.0.0/14 && ip4.dst == 10.128.0.0/14 allow 102 ip4.src == 10.128.0.0/14 && ip4.dst == 100.64.0.0/16 allow 102 pkt.mark == 42 allow 100 ip4.src == 10.130.2.11 reroute 100.88.0.6 But snat rules are duplicated I0123 14:18:22.777244 47011 client.go:835] Running 'oc --kubeconfig=/tmp/kubeconfig rsh -n openshift-ovn-kubernetes ovnkube-node-dl6r8 bash -c ovn-nbctl --format=csv --no-heading find nat | grep egressip-47021' I0123 14:18:28.495443 47011 cloud_egressip_ovn.go:2990] Defaulted container "ovn-controller" out of: ovn-controller, ovn-acl-logging, kube-rbac-proxy-node, kube-rbac-proxy-ovn-metrics, northd, nbdb, sbdb, ovnkube-controller, kubecfg-setup (init) 34884527-2d67-47c7-a2bd-f53f6bc37c90,[],[],"{ip-family=ip4, ""k8s.ovn.org/id""=""default-network-controller:EgressIP:egressip-47021_e2e-test-networking-jcjmrcip-xqdqr/test-rc-nc9b2:ip4"", ""k8s.ovn.org/name""=""egressip-47021_e2e-test-networking-jcjmrcip-xqdqr/test-rc-nc9b2"", ""k8s.ovn.org/owner-controller""=default-network-controller, ""k8s.ovn.org/owner-type""=EgressIP}","""10.0.14.52""",[],"""""",[],"""10.130.2.11""",k8s-ip-10-0-4-178.us-east-2.compute.internal,"""""","{stateless=""false""}",0,snat 1592e8de-e9b3-4817-b960-f014f47be332,[],[],"{ip-family=ip4, ""k8s.ovn.org/id""=""default-network-controller:EgressIP:egressip-47021_e2e-test-networking-tvjstjy9-ppzgs/test-rc-h9zgm:ip4"", ""k8s.ovn.org/name""=""egressip-47021_e2e-test-networking-tvjstjy9-ppzgs/test-rc-h9zgm"", ""k8s.ovn.org/owner-controller""=default-network-controller, ""k8s.ovn.org/owner-type""=EgressIP}","""10.0.14.52""",[],"""""",[],"""10.131.0.17""",k8s-ip-10-0-4-178.us-east-2.compute.internal,"""""","{stateless=""false""}",0,snat
After run, manually checking the env.
% oc get pods -n e2e-test-networking-jcjmrcip-xqdqr -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES test-rc-nc9b2 1/1 Running 0 17m 10.130.2.11 ip-10-0-64-214.us-east-2.compute.internal <none> <none> % oc get egressip NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS egressip-47021 10.0.14.52 ip-10-0-4-178.us-east-2.compute.internal 10.0.14.52 sh-5.1# ovn-nbctl --format=csv --no-heading find nat | grep egressip-47021 34884527-2d67-47c7-a2bd-f53f6bc37c90,[],[],"{ip-family=ip4, ""k8s.ovn.org/id""=""default-network-controller:EgressIP:egressip-47021_e2e-test-networking-jcjmrcip-xqdqr/test-rc-nc9b2:ip4"", ""k8s.ovn.org/name""=""egressip-47021_e2e-test-networking-jcjmrcip-xqdqr/test-rc-nc9b2"", ""k8s.ovn.org/owner-controller""=default-network-controller, ""k8s.ovn.org/owner-type""=EgressIP}","""10.0.14.52""",[],"""""",[],"""10.130.2.11""",k8s-ip-10-0-4-178.us-east-2.compute.internal,"""""","{stateless=""false""}",0,snat 1592e8de-e9b3-4817-b960-f014f47be332,[],[],"{ip-family=ip4, ""k8s.ovn.org/id""=""default-network-controller:EgressIP:egressip-47021_e2e-test-networking-tvjstjy9-ppzgs/test-rc-h9zgm:ip4"", ""k8s.ovn.org/name""=""egressip-47021_e2e-test-networking-tvjstjy9-ppzgs/test-rc-h9zgm"", ""k8s.ovn.org/owner-controller""=default-network-controller, ""k8s.ovn.org/owner-type""=EgressIP}","""10.0.14.52""",[],"""""",[],"""10.131.0.17""",k8s-ip-10-0-4-178.us-east-2.compute.internal,"""""","{stateless=""false""}",0,snat
3. Third run
Failed at lr-policy-list for egressIP missed
I0123 14:42:59.665372 47401 client.go:835] Running 'oc --kubeconfig=/tmp/kubeconfig rsh -n openshift-ovn-kubernetes ovnkube-node-85gbm bash -c ovn-nbctl lr-policy-list ovn_cluster_router | grep -v inport' I0123 14:43:05.083150 47401 cloud_egressip_ovn.go:2968] Defaulted container "ovn-controller" out of: ovn-controller, ovn-acl-logging, kube-rbac-proxy-node, kube-rbac-proxy-ovn-metrics, northd, nbdb, sbdb, ovnkube-controller, kubecfg-setup (init) Routing Policies 102 (ip4.src == $a8519615025667110816 || ip4.src == $a13607449821398607916) && ip4.dst == $a712973235162149816 allow pkt_mark=1008 102 ip4.src == 10.128.0.0/14 && ip4.dst == 10.128.0.0/14 allow 102 ip4.src == 10.128.0.0/14 && ip4.dst == 100.64.0.0/16 allow 102 pkt.mark == 42 allow
After run, manually checking the env,egressIP was not assigned to egress node
% oc get egressip
NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS
egressip-47021 10.0.7.233
The egress node does having the egress label
% oc get nodes --show-labels | grep egress
ip-10-0-4-178.us-east-2.compute.internal Ready worker 4h30m v1.31.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m6i.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-2,failure-domain.beta.kubernetes.io/zone=us-east-2a,k8s.ovn.org/egress-assignable=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-0-4-178.us-east-2.compute.internal,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.kubernetes.io/instance-type=m6i.xlarge,node.openshift.io/os_id=rhcos,topology.ebs.csi.aws.com/zone=us-east-2a,topology.k8s.aws/zone-id=use2-az1,topology.kubernetes.io/region=us-east-2,topology.kubernetes.io/zone=us-east-2a
From running log, egressIP was assigned to egress node before restarting ovn pod I0123 14:41:01.466637 47401 client.go:835] Running 'oc --kubeconfig=/tmp/kubeconfig get egressip egressip-47021 -ojsonpath={.status.items}' I0123 14:41:02.649955 47401 utils.go:1273] egressIPStatus: [{"egressIP":"10.0.21.18","node":"ip-10-0-4-178.us-east-2.compute.internal"}]
4. Fourth run
Failed at same point as third run I0123 15:01:31.048480 47820 client.go:835] Running 'oc --kubeconfig=/tmp/kubeconfig rsh -n openshift-ovn-kubernetes ovnkube-node-w2pr7 bash -c ovn-nbctl lr-policy-list ovn_cluster_router | grep -v inport' I0123 15:01:36.473679 47820 cloud_egressip_ovn.go:2968] Defaulted container "ovn-controller" out of: ovn-controller, ovn-acl-logging, kube-rbac-proxy-node, kube-rbac-proxy-ovn-metrics, northd, nbdb, sbdb, ovnkube-controller, kubecfg-setup (init) Routing Policies 102 (ip4.src == $a8519615025667110816 || ip4.src == $a13607449821398607916) && ip4.dst == $a712973235162149816 allow pkt_mark=1008 102 ip4.src == 10.128.0.0/14 && ip4.dst == 10.128.0.0/14 allow 102 ip4.src == 10.128.0.0/14 && ip4.dst == 100.64.0.0/16 allow 102 pkt.mark == 42 allow
Manually checking
% oc get egressip NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS egressip-47021 10.0.18.229 % oc get nodes --show-labels | grep egress ip-10-0-4-178.us-east-2.compute.internal Ready worker 4h47m v1.31.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m6i.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-2,failure-domain.beta.kubernetes.io/zone=us-east-2a,k8s.ovn.org/egress-assignable=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-0-4-178.us-east-2.compute.internal,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.kubernetes.io/instance-type=m6i.xlarge,node.openshift.io/os_id=rhcos,topology.ebs.csi.aws.com/zone=us-east-2a,topology.k8s.aws/zone-id=use2-az1,topology.kubernetes.io/region=us-east-2,topology.kubernetes.io/zone=us-east-2a % oc describe node ip-10-0-4-178.us-east-2.compute.internal | grep -C3 egress-ipconfig topology.k8s.aws/zone-id=use2-az1 topology.kubernetes.io/region=us-east-2 topology.kubernetes.io/zone=us-east-2a Annotations: cloud.network.openshift.io/egress-ipconfig: [{"interface":"eni-040e5f15c9a473467","ifaddr":{"ipv4":"10.0.0.0/19"},"capacity":{"ipv4":14,"ipv6":15}}] csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-002c83cf3533356c7"} k8s.ovn.org/bridge-egress-ips: []
3.
Actual results:
Expected results:
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
- internal CI failure
- customer issue / SD
- internal RedHat testing failure
If it is an internal RedHat testing failure:
- Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).
If it is a CI failure:
- Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
- Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
- Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
- When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
- If it's a connectivity issue,
- What is the srcNode, srcIP and srcNamespace and srcPodName?
- What is the dstNode, dstIP and dstNamespace and dstPodName?
- What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
If it is a customer / SD issue:
- Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
- Don’t presume that Engineering has access to Salesforce.
- Do presume that Engineering will access attachments through supportshell.
- Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
- Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
- If the issue is in a customer namespace then provide a namespace inspect.
- If it is a connectivity issue:
- What is the srcNode, srcNamespace, srcPodName and srcPodIP?
- What is the dstNode, dstNamespace, dstPodName and dstPodIP?
- What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
- Please provide the UTC timestamp networking outage window from must-gather
- Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
- If it is not a connectivity issue:
- Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.
- When showing the results from commands, include the entire command in the output.
- For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
- For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
- Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
- Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
- For guidance on using this template please see
OCPBUGS Template Training for Networking components
