-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.18
-
Important
-
None
-
Rejected
-
False
-
Description of problem:
Version-Release number of selected component (if applicable):
Pre-merge testing PR https://github.com/openshift/ovn-kubernetes/pull/2314
How reproducible:
Always
Steps to Reproduce:
1. Create a test namespace ns1
2. Create a layer3 CRD
% oc get userdefinednetwork -n ns1 -o yaml
apiVersion: v1
items:
- apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
creationTimestamp: "2024-10-11T10:10:36Z"
finalizers:
- k8s.ovn.org/user-defined-network-protection
generation: 1
name: udn-network
namespace: ns1
resourceVersion: "527896"
uid: 977d7248-39b1-4dfd-a437-44390d9fc884
spec:
layer3:
role: Primary
subnets:
- cidr: 10.200.0.0/16
hostSubnet: 24
topology: Layer3
status:
conditions:
- lastTransitionTime: "2024-10-11T10:10:36Z"
message: NetworkAttachmentDefinition has been created
reason: NetworkAttachmentDefinitionReady
status: "True"
type: NetworkReady
kind: List
metadata:
resourceVersion: ""
3. Create a pod and nodePort service in ns1
% oc get svc -n ns1 -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2024-10-11T10:11:49Z"
labels:
name: hello-pod
name: hello-pod
namespace: ns1
resourceVersion: "528359"
uid: f829a57b-2820-4308-9c69-bfc07383cc30
spec:
clusterIP: 172.30.234.28
clusterIPs:
- 172.30.234.28
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 30002
port: 27017
protocol: TCP
targetPort: 8080
selector:
name: hello-pod
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
kind: List
metadata:
resourceVersion: ""
% oc get pods -n ns1 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-pod 1/1 Running 0 5m1s 10.200.1.4 huirwang-1010c-w44lq-worker-b-d2k5v <none> <none>
% oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
huirwang-1010c-w44lq-master-0 Ready control-plane,master 24h v1.31.1 10.0.0.6 <none> Red Hat Enterprise Linux CoreOS 418.94.202410090804-0 5.14.0-427.40.1.el9_4.x86_64 cri-o://1.31.1-3.rhaos4.18.gitd8950b8.el9
huirwang-1010c-w44lq-master-1 Ready control-plane,master 24h v1.31.1 10.0.0.3 <none> Red Hat Enterprise Linux CoreOS 418.94.202410090804-0 5.14.0-427.40.1.el9_4.x86_64 cri-o://1.31.1-3.rhaos4.18.gitd8950b8.el9
huirwang-1010c-w44lq-master-2 Ready control-plane,master 24h v1.31.1 10.0.0.4 <none> Red Hat Enterprise Linux CoreOS 418.94.202410090804-0 5.14.0-427.40.1.el9_4.x86_64 cri-o://1.31.1-3.rhaos4.18.gitd8950b8.el9
huirwang-1010c-w44lq-worker-a-444b7 Ready worker 24h v1.31.1 10.0.128.2 <none> Red Hat Enterprise Linux CoreOS 418.94.202410090804-0 5.14.0-427.40.1.el9_4.x86_64 cri-o://1.31.1-3.rhaos4.18.gitd8950b8.el9
huirwang-1010c-w44lq-worker-b-d2k5v Ready worker 24h v1.31.1 10.0.128.3 <none> Red Hat Enterprise Linux CoreOS 418.94.202410090804-0 5.14.0-427.40.1.el9_4.x86_64 cri-o://1.31.1-3.rhaos4.18.gitd8950b8.el9
huirwang-1010c-w44lq-worker-c-l9dzj Ready worker 24h v1.31.1 10.0.128.4 <none> Red Hat Enterprise Linux CoreOS 418.94.202410090804-0 5.14.0-427.40.1.el9_4.x86_64 cri-o://1.31.1-3.rhaos4.18.gitd8950b8.el9
4. From a third node, to access the nodePort service, it succeded.
% oc debug node/huirwang-1010c-w44lq-master-0
Starting pod/huirwang-1010c-w44lq-master-0-debug-pcmch ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.0.6
If you don't see a command prompt, try pressing enter.
sh-5.1# curl 10.0.128.2:30002
Hello OpenShift!
From the node where the back-end pod was deployed to access nodePort service
Actual results:
Note able to access it.
% oc debug node/huirwang-1010c-w44lq-worker-a-444b7
Starting pod/huirwang-1010c-w44lq-worker-a-444b7-debug-t7qs4 ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.128.2
If you don't see a command prompt, try pressing enter.
sh-5.1# curl 10.0.128.2:30002 --connect-timeout 5
curl: (28) Connection timed out after 5000 milliseconds
Expected results:
Should be able to access nodePort service.
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
- internal CI failure
- customer issue / SD
- internal RedHat testing failure
If it is an internal RedHat testing failure:
- Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).
If it is a CI failure:
- Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
- Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
- Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
- When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
- If it's a connectivity issue,
- What is the srcNode, srcIP and srcNamespace and srcPodName?
- What is the dstNode, dstIP and dstNamespace and dstPodName?
- What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
If it is a customer / SD issue:
- Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
- Don’t presume that Engineering has access to Salesforce.
- Do presume that Engineering will access attachments through supportshell.
- Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
- Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
- If the issue is in a customer namespace then provide a namespace inspect.
- If it is a connectivity issue:
- What is the srcNode, srcNamespace, srcPodName and srcPodIP?
- What is the dstNode, dstNamespace, dstPodName and dstPodIP?
- What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
- Please provide the UTC timestamp networking outage window from must-gather
- Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
- If it is not a connectivity issue:
- Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.
- When showing the results from commands, include the entire command in the output.
- For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
- For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
- Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
- Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
- For guidance on using this template please see
OCPBUGS Template Training for Networking components
