[ARO] egressIP cannot work on aro azure cluster

4.10.54

Always

1. Patch egressIP to one worker node
% oc get hostsubnet
NAME                                              HOST                                              HOST IP     SUBNET          EGRESS CIDRS   EGRESS IPS
huirwang-aro1-4rcjz-master-0                      huirwang-aro1-4rcjz-master-0                      10.0.0.8    10.129.0.0/23                  
huirwang-aro1-4rcjz-master-1                      huirwang-aro1-4rcjz-master-1                      10.0.0.10   10.130.0.0/23                  
huirwang-aro1-4rcjz-master-2                      huirwang-aro1-4rcjz-master-2                      10.0.0.9    10.128.0.0/23                  
huirwang-aro1-4rcjz-worker-northcentralus-6tn7n   huirwang-aro1-4rcjz-worker-northcentralus-6tn7n   10.0.2.5    10.129.2.0/23                  
huirwang-aro1-4rcjz-worker-northcentralus-rgpnl   huirwang-aro1-4rcjz-worker-northcentralus-rgpnl   10.0.2.6    10.131.0.0/23                  ["10.0.2.13"]
huirwang-aro1-4rcjz-worker-northcentralus-z7dvh   huirwang-aro1-4rcjz-worker-northcentralus-z7dvh   10.0.2.4    10.128.2.0/23          

2. Create namespace and a pod in it, patch egressIP to it
% oc get netnamespace 46710-upgrade-ns                                                                 
NAME               NETID     EGRESS IPS
46710-upgrade-ns   8856810   ["10.0.2.13"]
3.

% oc get cloudprivateipconfig -o yaml 
apiVersion: v1
items:
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    creationTimestamp: "2023-07-04T08:53:22Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.2.13
    resourceVersion: "38148"
    uid: c5b75cd1-773c-42f6-82de-f7624f333872
  spec:
    node: huirwang-aro1-4rcjz-worker-northcentralus-rgpnl
  status:
    conditions:
    - lastTransitionTime: "2023-07-04T08:58:55Z"
      message: 'Error processing cloud assignment request, err: <nil>'
      observedGeneration: 1
      reason: CloudResponseError
      status: "False"
      type: Assigned
    node: huirwang-aro1-4rcjz-worker-northcentralus-rgpnl
kind: List
metadata:
  resourceVersion: ""

 % oc logs cloud-network-config-controller-5995968998-2l9dw -n openshift-cloud-network-config-controller
W0704 08:28:08.604224       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0704 08:28:08.605897       1 leaderelection.go:248] attempting to acquire leader lease openshift-cloud-network-config-controller/cloud-network-config-controller-lock...
E0704 08:28:38.612612       1 leaderelection.go:330] error retrieving resource lock openshift-cloud-network-config-controller/cloud-network-config-controller-lock: Get "https://api-int.fma82m9s.northcentralus.aroapp.io:6443/api/v1/namespaces/openshift-cloud-network-config-controller/configmaps/cloud-network-config-controller-lock": dial tcp: i/o timeout
I0704 08:29:23.515143       1 leaderelection.go:258] successfully acquired lease openshift-cloud-network-config-controller/cloud-network-config-controller-lock
I0704 08:29:23.516229       1 controller.go:88] Starting node controller
I0704 08:29:23.516249       1 controller.go:91] Waiting for informer caches to sync for node workqueue
I0704 08:29:23.516268       1 controller.go:88] Starting secret controller
I0704 08:29:23.516273       1 controller.go:91] Waiting for informer caches to sync for secret workqueue
I0704 08:29:23.516289       1 controller.go:88] Starting cloud-private-ip-config controller
I0704 08:29:23.516293       1 controller.go:91] Waiting for informer caches to sync for cloud-private-ip-config workqueue
I0704 08:29:23.526407       1 controller.go:182] Assigning key: huirwang-aro1-4rcjz-master-1 to node workqueue
I0704 08:29:23.526428       1 controller.go:182] Assigning key: huirwang-aro1-4rcjz-master-2 to node workqueue
I0704 08:29:23.526438       1 controller.go:182] Assigning key: huirwang-aro1-4rcjz-master-0 to node workqueue
I0704 08:29:23.616805       1 controller.go:96] Starting cloud-private-ip-config workers
I0704 08:29:23.616892       1 controller.go:96] Starting node workers
I0704 08:29:23.616894       1 controller.go:102] Started cloud-private-ip-config workers
I0704 08:29:23.617000       1 controller.go:96] Starting secret workers
I0704 08:29:23.617209       1 controller.go:102] Started node workers
I0704 08:29:23.617245       1 controller.go:102] Started secret workers
E0704 08:29:24.011675       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.032967       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.047501       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.061145       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.080864       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.105632       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.124561       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.143735       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.163432       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.193602       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.219350       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.233470       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.281336       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.313263       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.321940       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.406418       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.437888       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.448434       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.615179       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.653841       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.654347       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:24.996557       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:25.024158       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:25.041019       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:25.680147       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:25.708946       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:25.725800       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:27.001706       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:27.044504       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:27.050471       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:29.609735       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:29.651282       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:29.665269       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:34.763052       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:34.829348       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:34.831547       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:45.046486       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:45.120839       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:29:45.136157       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:30:05.603540       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:30:05.647939       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:30:05.671575       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:30:46.600243       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:30:46.671021       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:30:46.692343       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
E0704 08:32:08.569773       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-1': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-1, err: no retrievable network interface, requeuing in node workqueue
E0704 08:32:08.638076       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-2': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-2, err: no retrievable network interface, requeuing in node workqueue
E0704 08:32:08.667971       1 controller.go:165] error syncing 'huirwang-aro1-4rcjz-master-0': error retrieving the private IP configuration for node: huirwang-aro1-4rcjz-master-0, err: no retrievable network interface, requeuing in node workqueue
I0704 08:34:32.068644       1 controller.go:182] Assigning key: huirwang-aro1-4rcjz-worker-northcentralus-rgpnl to node workqueue
I0704 08:34:32.356023       1 node_controller.go:106] Setting annotation: 'cloud.network.openshift.io/egress-ipconfig: [{"interface":"huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic","ifaddr":{"ipv4":"10.0.2.0/23"},"capacity":{"ip":255}}]' on node: huirwang-aro1-4rcjz-worker-northcentralus-rgpnl
I0704 08:34:32.405751       1 controller.go:160] Dropping key 'huirwang-aro1-4rcjz-worker-northcentralus-rgpnl' from the node workqueue
I0704 08:34:52.462704       1 controller.go:160] Dropping key 'huirwang-aro1-4rcjz-master-1' from the node workqueue
I0704 08:34:52.538212       1 controller.go:160] Dropping key 'huirwang-aro1-4rcjz-master-2' from the node workqueue
I0704 08:34:52.563578       1 controller.go:160] Dropping key 'huirwang-aro1-4rcjz-master-0' from the node workqueue
E0704 08:36:06.410447       1 leaderelection.go:330] error retrieving resource lock openshift-cloud-network-config-controller/cloud-network-config-controller-lock: the server was unable to return a response in the time allotted, but may still be processing the request (get configmaps cloud-network-config-controller-lock)
I0704 08:37:04.547977       1 controller.go:182] Assigning key: huirwang-aro1-4rcjz-worker-northcentralus-z7dvh to node workqueue
I0704 08:37:04.821736       1 node_controller.go:106] Setting annotation: 'cloud.network.openshift.io/egress-ipconfig: [{"interface":"huirwang-aro1-4rcjz-worker-northcentralus-z7dvh-nic","ifaddr":{"ipv4":"10.0.2.0/23"},"capacity":{"ip":255}}]' on node: huirwang-aro1-4rcjz-worker-northcentralus-z7dvh
I0704 08:37:04.867217       1 controller.go:160] Dropping key 'huirwang-aro1-4rcjz-worker-northcentralus-z7dvh' from the node workqueue
I0704 08:37:05.619660       1 controller.go:182] Assigning key: huirwang-aro1-4rcjz-worker-northcentralus-6tn7n to node workqueue
I0704 08:37:05.813949       1 node_controller.go:106] Setting annotation: 'cloud.network.openshift.io/egress-ipconfig: [{"interface":"huirwang-aro1-4rcjz-worker-northcentralus-6tn7n-nic","ifaddr":{"ipv4":"10.0.2.0/23"},"capacity":{"ip":255}}]' on node: huirwang-aro1-4rcjz-worker-northcentralus-6tn7n
I0704 08:37:05.877894       1 controller.go:160] Dropping key 'huirwang-aro1-4rcjz-worker-northcentralus-6tn7n' from the node workqueue
I0704 08:53:22.280443       1 controller.go:182] Assigning key: 10.0.2.13 to cloud-private-ip-config workqueue
I0704 08:53:22.294451       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
I0704 08:53:22.306269       1 controller.go:182] Assigning key: 10.0.2.13 to cloud-private-ip-config workqueue
I0704 08:53:22.306488       1 cloudprivateipconfig_controller.go:295] Adding finalizer to CloudPrivateIPConfig: "10.0.2.13"
E0704 08:53:23.809945       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:23.828663       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:24.147884       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:24.159920       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:24.446737       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:24.451694       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:24.740672       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:24.745754       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:25.039477       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:25.046513       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:25.322954       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:25.329564       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:25.680950       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:25.789308       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:26.257667       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:26.392444       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:26.890991       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:26.999420       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:27.442533       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:30.020362       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:30.277621       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:35.405079       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:35.763560       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:53:46.015387       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:53:46.257603       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:54:06.746283       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:54:07.046902       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:54:48.019662       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:54:48.305141       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:56:10.233551       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
E0704 08:56:10.492018       1 controller.go:165] error syncing '10.0.2.13': error assigning CloudPrivateIPConfig: "10.0.2.13" to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/outboundRules/outbound-rule-v4 cannot be used with Backend Address Pool /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/loadBalancers/huirwang-aro1-4rcjz/backendAddressPools/huirwang-aro1-4rcjz that contains Secondary IPConfig /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/aro-fma82m9s/providers/Microsoft.Network/networkInterfaces/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl-nic/ipConfigurations/huirwang-aro1-4rcjz-worker-northcentralus-rgpnl_10.0.2.13" Details=[], requeuing in cloud-private-ip-config workqueue
I0704 08:58:54.344340       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.2.13" will be added to node: "huirwang-aro1-4rcjz-worker-northcentralus-rgpnl"
I0704 08:58:55.793584       1 controller.go:160] Dropping key '10.0.2.13' from the cloud-private-ip-config workqueue

EgressIP works well