-
Bug
-
Resolution: Done
-
Undefined
-
None
-
4.12
-
None
-
Important
-
None
-
SDN Sprint 227
-
1
-
Rejected
-
False
-
-
NA
Description of problem:
Version-Release number of selected component (if applicable):
4.12.0-0.nightly-2022-08-22-143022
How reproducible:
Tested this scenario twice and happened twice
Steps to Reproduce:
1. Label 3 worker nodes as egress nodes
2. Create ns test and add label name=qe
3. Create first egressip object
$ oc get egressip
NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS
egressip-2 192.168.1.20 huirwang-0823a-2p9b4-worker-2 192.168.1.20
4. Create second egressip which is same egressIP as above.
$ oc get egressip
NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS
egressip-1 192.168.1.20
egressip-2 192.168.1.20 huirwang-0823a-2p9b4-worker-2 192.168.1.20
egressip-1 was not assigned to node, this is as expected.
5. Edit egressip-1's IP from 192.168.1.20 to 192.168.1.21, wait some time
Actual results:
192.168.1.21 was not assigned correctly.
$ oc get egressip
NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS
egressip-1 192.168.1.21
egressip-2 192.168.1.20 huirwang-0823a-2p9b4-worker-2 192.168.1.20
$ oc get cloudprivateipconfig
NAME AGE
192.168.1.20 23m
192.168.1.21 23s
$ oc get cloudprivateipconfig 192.168.1.21 -o yaml
apiVersion: cloud.network.openshift.io/v1
kind: CloudPrivateIPConfig
metadata:
annotations:
k8s.ovn.org/egressip-owner-ref: egressip-1
creationTimestamp: "2022-08-23T02:40:50Z"
finalizers:
- cloudprivateipconfig.cloud.network.openshift.io/finalizer
generation: 1
name: 192.168.1.21
resourceVersion: "59775"
uid: d24f1c2b-4c96-44a4-9f57-7b1706fb9dda
spec:
node: huirwang-0823a-2p9b4-worker-1
status:
conditions:
- lastTransitionTime: "2022-08-23T02:40:53Z"
message: IP address successfully added
observedGeneration: 1
reason: CloudResponseSuccess
status: "True"
type: Assigned
node: huirwang-0823a-2p9b4-worker-1
$ oc get nodes
NAME STATUS ROLES AGE VERSION
huirwang-0823a-2p9b4-master-0 Ready control-plane,master 89m v1.24.0+ed93380
huirwang-0823a-2p9b4-master-1 Ready control-plane,master 87m v1.24.0+ed93380
huirwang-0823a-2p9b4-master-2 Ready control-plane,master 85m v1.24.0+ed93380
huirwang-0823a-2p9b4-worker-0 Ready worker 74m v1.24.0+ed93380
huirwang-0823a-2p9b4-worker-1 Ready worker 74m v1.24.0+ed93380
huirwang-0823a-2p9b4-worker-2 Ready worker 74m v1.24.0+ed93380
Check ovnkube-master log, unable to create logical router policy
16289 I0823 02:56:21.362947 1 obj_retry.go:1085] *v1.CloudPrivateIPConfig retry update failed for 192.168.1.21, will try again later: unable to create logical router policy, err: erro r creating logical router policy {UUID:u2596996643 Action:reroute ExternalIDs:map[name:egressip-1] Match:ip4.src == 10.129.2.17 Nexthop:<nil> Nexthops:[100.64.0.6] Options:map[] Prior ity:100} on router ovn_cluster_router: error in transact with ops [{Op:wait Table:Logical_Router_Policy Row:map[] Rows:[map[match:ip4.src == 10.129.2.17 priority:100]] Columns:[priori ty match] Mutations:[] Timeout:0xc001ea3120 Where:[where column _uuid == {bcc02607-91d1-4a37-9512-888b79636cbb}] Until:!= Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:insert Table:Logical_Router_Policy Row:map[action:reroute external_ids:{GoMap:map[name:egressip-1]} match:ip4.src == 10.129.2.17 nexthops:{GoSet:[100.64.0.6]} priority:100] Rows:[] Columns:[ ] Mutations:[] Timeout:<nil> Where:[] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:u2596996643} {Op:mutate Table:Logical_Router Row:map[] Rows:[] Columns:[] Mutations:[{Colu mn:policies Mutator:insert Value:{GoSet:[{GoUUID:u2596996643}]}}] Timeout:<nil> Where:[where column _uuid == {1c489a96-469b-4a57-9eb8-06dccae068d5}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:0 Error:timed out Details:"where" clause test failed UUID:{GoUUID:} Rows:[]} {Count:0 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:0 Error: D etails: UUID:{GoUUID:} Rows:[]}] and errors [timed out: "where" clause test failed]: 1 ovsdb operations failed 16290 I0823 02:56:21.404980 1 master.go:1341] Adding or Updating Node "huirwang-0823a-2p9b4-worker-1" 16291 I0823 02:56:32.896941 1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.NetworkPolicy total 11 items received 16292 I0823 02:56:41.118865 1 master.go:1341] Adding or Updating Node "huirwang-0823a-2p9b4-worker-0" 16293 I0823 02:56:43.098360 1 master.go:1341] Adding or Updating Node "huirwang-0823a-2p9b4-worker-2" 16294 I0823 02:56:46.899630 1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 23 items received 16295 I0823 02:56:49.902013 1 transact.go:41] Configuring OVN: [{Op:update Table:NB_Global Row:map[options:{GoMap:map[controller_event:true e2e_timestamp:1661223409 mac_prefix:d2:f0:9 2 max_tunid:16711680 northd_internal_version:22.06.1-20.23.0-63.4 northd_probe_interval:10000 svc_monitor_mac:2a:d8:51:4f:00:e2 use_logical_dp_groups:true]}] Rows:[] Columns:[] Mutati ons:[] Timeout:<nil> Where:[where column _uuid == {c9788c58-83f6-4f50-a779-23a8d9e7dd8a}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] 16296 I0823 02:56:49.902184 1 client.go:781] "msg"="transacting operations" "database"="OVN_Northbound" "operations"="[{Op:update Table:NB_Global Row:map[options:{GoMap:map[controll er_event:true e2e_timestamp:1661223409 mac_prefix:d2:f0:92 max_tunid:16711680 northd_internal_version:22.06.1-20.23.0-63.4 northd_probe_interval:10000 svc_monitor_mac:2a:d8:51:4f:00:e 2 use_logical_dp_groups:true]}] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {c9788c58-83f6-4f50-a779-23a8d9e7dd8a}] Until: Durable:<nil> Comment:<nil> L ock:<nil> UUIDName:}]" 16297 W0823 02:56:51.353724 1 obj_retry.go:1031] Dropping retry entry for *v1.CloudPrivateIPConfig 192.168.1.21: exceeded number of failed attempts
Expected results:
The egressIP should be updated correctly.
Additional info:
