As a day 2 operation, when workers are added to SNO cluster, they falls into NOTREADY state due to ovnkube complains " no annotation found on node for subnet: could not find "k8s.ovn.org/node-subnets" annotation`

Must-gather: http://shell.lab.bos.redhat.com/~anusaxen/must-gather.local.601424534302218678/

4.13.0-0.nightly-2023-03-11-033820

Always

1.Install SNO cluster
2.Add worker nodes as a part of day 2 operation
3.

Workers falls into NOTREADY state

Workers should be installed without any issues

$ oc get nodes
NAME                                                  STATUS     ROLES                         AGE     VERSION
master-00.yanyang-34777.qe.devcluster.openshift.com   Ready      control-plane,master,worker   7h21m   v1.26.2+bc894ae
worker-00.yanyang-34777.qe.devcluster.openshift.com   NotReady   worker                        6h40m   v1.26.2+bc894ae
worker-01.yanyang-34777.qe.devcluster.openshift.com   NotReady   worker                        6h40m   v1.26.2+bc894ae


$oc get pods -n openshift-ovn-kubernetes
NAME                   READY   STATUS             RESTARTS        AGE
ovnkube-master-xk6jk   6/6     Running            0               7h20m
ovnkube-node-plvw8     5/5     Running            1 (7h18m ago)   7h20m
ovnkube-node-s28hs     4/5     Running            59 (60s ago)    6h39m
ovnkube-node-x6jt5     4/5     CrashLoopBackOff   63 (28s ago)    6h39m

one of worker's ovnkube-node logs:

I0317 14:28:07.866312  100574 default_node_network_controller.go:428] OVN Kube Node initialization, Mode: full
I0317 14:28:07.867715  100574 config.go:1366] Exec: /usr/bin/ovs-vsctl --timeout=15 del-ssl
I0317 14:28:07.870033  100574 ovs.go:203] Exec(2): stdout: ""
I0317 14:28:07.870047  100574 ovs.go:204] Exec(2): stderr: ""
I0317 14:28:07.870052  100574 node_network_controller_manager.go:277] CheckForStaleOVSInternalPorts took 3.787473ms
I0317 14:28:07.870069  100574 ovs.go:200] Exec(3): /usr/bin/ovs-vsctl --timeout=15 --columns=name,external_ids --data=bare --no-headings --format=csv find Interface external_ids:sandbox!="" external_ids:vf-netde
v-name!=""
I0317 14:28:07.871889  100574 config.go:1366] Exec: /usr/bin/ovs-vsctl --timeout=15 set-ssl /ovn-cert/tls.key /ovn-cert/tls.crt /ovn-ca/ca-bundle.crt
I0317 14:28:07.873445  100574 ovs.go:203] Exec(3): stdout: ""
I0317 14:28:07.873455  100574 ovs.go:204] Exec(3): stderr: ""
I0317 14:28:07.877090  100574 config.go:1366] Exec: /usr/bin/ovs-vsctl --timeout=15 set Open_vSwitch . external_ids:ovn-remote="ssl:147.28.129.55:9642"
I0317 14:28:07.880502  100574 ovs.go:200] Exec(4): /usr/bin/ovs-vsctl --timeout=15 set Open_vSwitch . external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=147.75.51.17 external_ids:ovn-remote-probe-inter
val=180000 external_ids:ovn-openflow-probe-interval=180 other_config:bundle-idle-timeout=180 external_ids:hostname="worker-00.yanyang-34777.qe.devcluster.openshift.com" external_ids:ovn-monitor-all=true external
_ids:ovn-ofctrl-wait-before-clear=0 external_ids:ovn-enable-lflow-cache=true external_ids:ovn-memlimit-lflow-cache-kb=1048576
I0317 14:28:07.883374  100574 ovs.go:203] Exec(4): stdout: ""
I0317 14:28:07.883386  100574 ovs.go:204] Exec(4): stderr: ""
I0317 14:28:07.883396  100574 ovs.go:200] Exec(5): /usr/bin/ovs-vsctl --timeout=15 -- clear bridge br-int netflow -- clear bridge br-int sflow -- clear bridge br-int ipfix
I0317 14:28:07.886379  100574 ovs.go:203] Exec(5): stdout: ""
I0317 14:28:07.886386  100574 ovs.go:204] Exec(5): stderr: ""
I0317 14:28:07.887519  100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov
n.org/node-subnets" annotation
I0317 14:28:08.389537  100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov
n.org/node-subnets" annotation
I0317 14:28:08.889024  100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov
n.org/node-subnets" annotation
I0317 14:28:09.389559  100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov
n.org/node-subnets" annotation
I0317 14:28:09.889211  100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov
n.org/node-subnets" annotation
I0317 14:28:10.388746  100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov
n.org/node-subnets" annotation
I0317 14:28:10.889673  100574 default_node_network_controller.go:471] Waiting for node worker-00.yanyang-34777.qe.devcluster.openshift.com to start, no annotation found on node for subnet: could not find "k8s.ov
n.org/node-subnets" annotation