-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.18
-
Important
-
None
-
Proposed
-
False
-
Description of problem:
When running with 216 namespace, each having its UDN, we see that some of the pods in the namespaces are stuck in ContainerCreating. Upon looking further it seems that the nodes assigned to these pods no do have UDN in k8s.ovn.org/node-mgmt-port-mac-addresses in node annotations.
Example of the error
I1028 23:07:15.743936 3102 obj_retry.go:296] Retry object setup: *v1.Node ip-10-0-89-135.us-west-2.compute.internal I1028 23:07:15.744059 3102 obj_retry.go:358] Adding new object: *v1.Node ip-10-0-89-135.us-west-2.compute.internal I1028 23:07:15.744098 3102 secondary_layer3_network_controller.go:625] Adding or Updating Node "ip-10-0-89-135.us-west-2.compute.internal" for network udn-density-l3-pods-214.l3-network-214 E1028 23:07:15.779263 3102 secondary_layer3_network_controller.go:131] Node add failed for ip-10-0-89-135.us-west-2.compute.internal, will try again later: nodeAdd: error adding node "ip-10-0-89-135.us-west-2.compute.internal" for network udn-density-l3-pods-214.l3-network-214: failed to build UDN masquerade SNATs for network "udn-density-l3-pods-214.l3-network-214" on node "ip-10-0-89-135.us-west-2.compute.internal", err: failed to parse mac address annotation for network "udn-density-l3-pods-214.l3-network-214" on node "ip-10-0-89-135.us-west-2.compute.internal", err: node "ip-10-0-89-135.us-west-2.compute.internal" has no "k8s.ovn.org/node-mgmt-port-mac-addresses" annotation for network udn-density-l3-pods-214.l3-network-214 I1028 23:07:15.779370 3102 obj_retry.go:370] Retry add failed for *v1.Node ip-10-0-89-135.us-west-2.compute.internal, will try again later: nodeAdd: error adding node "ip-10-0-89-135.us-west-2.compute.internal" for network udn-density-l3-pods-214.l3-network-214: failed to build UDN masquerade SNATs for network "udn-density-l3-pods-214.l3-network-214" on node "ip-10-0-89-135.us-west-2.compute.internal", err: failed to parse mac address annotation for network "udn-density-l3-pods-214.l3-network-214" on node "ip-10-0-89-135.us-west-2.compute.internal", err: node "ip-10-0-89-135.us-west-2.compute.internal" has no "k8s.ovn.org/node-mgmt-port-mac-addresses" annotation for network udn-density-l3-pods-214.l3-network-214
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-10-28-052434
How reproducible: Once for now with the above nightly
Steps to Reproduce:
1. Run UDN density workload at 24 nodes with 216 iterations
Actual results:
Test does not complete as all pods are not attached to a network
Expected results:
Test completes with all pods getting uits UDN IP
