Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
4.13.0
-
Important
-
No
-
SDN Sprint 233
-
1
-
Approved
-
False
-
Description
Description of problem:
As a day 2 operation, when workers are added to SNO cluster, they falls into NOTREADY state due to ovnkube complains " no annotation found on node for subnet: could not find "k8s.ovn.org/node-subnets" annotation` Must-gather: http://shell.lab.bos.redhat.com/~anusaxen/must-gather.local.601424534302218678/
Version-Release number of selected component (if applicable):
4.13.0-0.nightly-2023-03-11-033820
How reproducible:
Always
Steps to Reproduce:
1.Install SNO cluster 2.Add worker nodes as a part of day 2 operation 3.
Actual results:
Workers falls into NOTREADY state
Expected results:
Workers should be installed without any issues
Additional info:
$ oc get nodes NAME STATUS ROLES AGE VERSION master-00.yanyang-34777.qe.devcluster.openshift.com Ready control-plane,master,worker 7h21m v1.26.2+bc894ae worker-00.yanyang-34777.qe.devcluster.openshift.com NotReady worker 6h40m v1.26.2+bc894ae worker-01.yanyang-34777.qe.devcluster.openshift.com NotReady worker 6h40m v1.26.2+bc894ae $oc get pods -n openshift-ovn-kubernetes NAME READY STATUS RESTARTS AGE ovnkube-master-xk6jk 6/6 Running 0 7h20m ovnkube-node-plvw8 5/5 Running 1 (7h18m ago) 7h20m ovnkube-node-s28hs 4/5 Running 59 (60s ago) 6h39m ovnkube-node-x6jt5 4/5 CrashLoopBackOff 63 (28s ago) 6h39m one of worker's ovnkube-node logs: I0317 14:28:07.866312 100574 default_node_network_controller.go:428] OVN Kube Node initialization, Mode: full I0317 14:28:07.867715 100574 config.go:1366] Exec: /usr/bin/ovs-vsctl --timeout=15 del-ssl I0317 14:28:07.870033 100574 ovs.go:203] Exec(2): stdout: "" I0317 14:28:07.870047 100574 ovs.go:204] Exec(2): stderr: "" I0317 14:28:07.870052 100574 node_network_controller_manager.go:277] CheckForStaleOVSInternalPorts took 3.787473ms I0317 14:28:07.870069 100574 ovs.go:200] Exec(3): /usr/bin/ovs-vsctl --timeout=15 --columns=name,external_ids --data=bare --no-headings --format=csv find Interface external_ids:sandbox!="" external_ids:vf-netde v-name!="" I0317 14:28:07.871889 100574 config.go:1366] Exec: /usr/bin/ovs-vsctl --timeout=15 set-ssl /ovn-cert/tls.key /ovn-cert/tls.crt /ovn-ca/ca-bundle.crt I0317 14:28:07.873445 100574 ovs.go:203] Exec(3): stdout: "" I0317 14:28:07.873455 100574 ovs.go:204] Exec(3): stderr: "" I0317 14:28:07.877090 100574 config.go:1366] Exec: /usr/bin/ovs-vsctl --timeout=15 set Open_vSwitch . external_ids:ovn-remote="ssl:147.28.129.55:9642" I0317 14:28:07.880502 100574 ovs.go:200] Exec(4): /usr/bin/ovs-vsctl --timeout=15 set Open_vSwitch . external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=147.75.51.17 external_ids:ovn-remote-probe-inter val=180000 external_ids:ovn-openflow-probe-interval=180 other_config:bundle-idle-timeout=180 external_ids:hostname="worker-00.yanyang-34777.qe.devcluster.openshift.com" external_ids:ovn-monitor-all=true external _ids:ovn-ofctrl-wait-before-clear=0 external_ids:ovn-enable-lflow-cache=true external_ids:ovn-memlimit-lflow-cache-kb=1048576 I0317 14:28:07.883374 100574 ovs.go:203] Exec(4): stdout: "" I0317 14:28:07.883386 100574 ovs.go:204] Exec(4): stderr: "" I0317 14:28:07.883396 100574 ovs.go:200] Exec(5): /usr/bin/ovs-vsctl --timeout=15 -- clear bridge br-int netflow -- clear bridge br-int sflow -- clear bridge br-int ipfix I0317 14:28:07.886379 100574 ovs.go:203] Exec(5): stdout: "" I0317 14:28:07.886386 100574 ovs.go:204] Exec(5): stderr: "" I0317 14:28:07.887519 100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov n.org/node-subnets" annotation I0317 14:28:08.389537 100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov n.org/node-subnets" annotation I0317 14:28:08.889024 100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov n.org/node-subnets" annotation I0317 14:28:09.389559 100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov n.org/node-subnets" annotation I0317 14:28:09.889211 100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov n.org/node-subnets" annotation I0317 14:28:10.388746 100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov n.org/node-subnets" annotation I0317 14:28:10.889673 100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov n.org/node-subnets" annotation
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-
