Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-426

[OSP][OVN]unable to create logical router policy for egressIP after update duplicate IP to uniq one

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • 4.12
    • None
    • Important
    • SDN Sprint 227
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • NA

      Description of problem:

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-08-22-143022 

      How reproducible:

      Tested this scenario twice and happened twice

      Steps to Reproduce:
      1. Label 3 worker nodes as egress nodes

      2. Create ns test and add label name=qe

      3. Create first egressip object

      $ oc get egressip
      NAME         EGRESSIPS      ASSIGNED NODE                   ASSIGNED EGRESSIPS
      egressip-2   192.168.1.20   huirwang-0823a-2p9b4-worker-2   192.168.1.20

      4. Create second egressip which is same egressIP as above.

      $ oc get egressip
      NAME         EGRESSIPS      ASSIGNED NODE                   ASSIGNED EGRESSIPS
      egressip-1   192.168.1.20                                   
      egressip-2   192.168.1.20   huirwang-0823a-2p9b4-worker-2   192.168.1.20

      egressip-1 was not assigned to node, this is as expected.

      5. Edit egressip-1's IP from 192.168.1.20  to 192.168.1.21, wait some time

       

      Actual results:

      192.168.1.21  was not assigned correctly.

      $ oc get egressip
      NAME         EGRESSIPS      ASSIGNED NODE                   ASSIGNED EGRESSIPS
      egressip-1   192.168.1.21                                   
      egressip-2   192.168.1.20   huirwang-0823a-2p9b4-worker-2   192.168.1.20

      $ oc get cloudprivateipconfig
      NAME           AGE
      192.168.1.20   23m
      192.168.1.21   23s

      $ oc get cloudprivateipconfig 192.168.1.21 -o yaml
      apiVersion: cloud.network.openshift.io/v1
      kind: CloudPrivateIPConfig
      metadata:
        annotations:
          k8s.ovn.org/egressip-owner-ref: egressip-1
        creationTimestamp: "2022-08-23T02:40:50Z"
        finalizers:
        - cloudprivateipconfig.cloud.network.openshift.io/finalizer
        generation: 1
        name: 192.168.1.21
        resourceVersion: "59775"
        uid: d24f1c2b-4c96-44a4-9f57-7b1706fb9dda
      spec:
        node: huirwang-0823a-2p9b4-worker-1
      status:
        conditions:
        - lastTransitionTime: "2022-08-23T02:40:53Z"
          message: IP address successfully added
          observedGeneration: 1
          reason: CloudResponseSuccess
          status: "True"
          type: Assigned
        node: huirwang-0823a-2p9b4-worker-1

       

      $ oc get nodes
      NAME                            STATUS   ROLES                  AGE   VERSION
      huirwang-0823a-2p9b4-master-0   Ready    control-plane,master   89m   v1.24.0+ed93380
      huirwang-0823a-2p9b4-master-1   Ready    control-plane,master   87m   v1.24.0+ed93380
      huirwang-0823a-2p9b4-master-2   Ready    control-plane,master   85m   v1.24.0+ed93380
      huirwang-0823a-2p9b4-worker-0   Ready    worker                 74m   v1.24.0+ed93380
      huirwang-0823a-2p9b4-worker-1   Ready    worker                 74m   v1.24.0+ed93380
      huirwang-0823a-2p9b4-worker-2   Ready    worker                 74m   v1.24.0+ed93380

       

      Check ovnkube-master log, unable to create logical router policy

      16289 I0823 02:56:21.362947       1 obj_retry.go:1085] *v1.CloudPrivateIPConfig retry update failed for 192.168.1.21, will try again later: unable to create logical router policy, err: erro      r creating logical router policy {UUID:u2596996643 Action:reroute ExternalIDs:map[name:egressip-1] Match:ip4.src == 10.129.2.17 Nexthop:<nil> Nexthops:[100.64.0.6] Options:map[] Prior      ity:100} on router ovn_cluster_router: error in transact with ops [{Op:wait Table:Logical_Router_Policy Row:map[] Rows:[map[match:ip4.src == 10.129.2.17 priority:100]] Columns:[priori      ty match] Mutations:[] Timeout:0xc001ea3120 Where:[where column _uuid == {bcc02607-91d1-4a37-9512-888b79636cbb}] Until:!= Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:insert       Table:Logical_Router_Policy Row:map[action:reroute external_ids:{GoMap:map[name:egressip-1]} match:ip4.src == 10.129.2.17 nexthops:{GoSet:[100.64.0.6]} priority:100] Rows:[] Columns:[      ] Mutations:[] Timeout:<nil> Where:[] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:u2596996643} {Op:mutate Table:Logical_Router Row:map[] Rows:[] Columns:[] Mutations:[{Colu      mn:policies Mutator:insert Value:{GoSet:[{GoUUID:u2596996643}]}}] Timeout:<nil> Where:[where column _uuid == {1c489a96-469b-4a57-9eb8-06dccae068d5}] Until: Durable:<nil> Comment:<nil>       Lock:<nil> UUIDName:}] results [{Count:0 Error:timed out Details:"where" clause test failed UUID:{GoUUID:} Rows:[]} {Count:0 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:0 Error: D      etails: UUID:{GoUUID:} Rows:[]}] and errors [timed out: "where" clause test failed]: 1 ovsdb operations failed
      16290 I0823 02:56:21.404980       1 master.go:1341] Adding or Updating Node "huirwang-0823a-2p9b4-worker-1"
      16291 I0823 02:56:32.896941       1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.NetworkPolicy total 11 items received
      16292 I0823 02:56:41.118865       1 master.go:1341] Adding or Updating Node "huirwang-0823a-2p9b4-worker-0"
      16293 I0823 02:56:43.098360       1 master.go:1341] Adding or Updating Node "huirwang-0823a-2p9b4-worker-2"
      16294 I0823 02:56:46.899630       1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 23 items received
      16295 I0823 02:56:49.902013       1 transact.go:41] Configuring OVN: [{Op:update Table:NB_Global Row:map[options:{GoMap:map[controller_event:true e2e_timestamp:1661223409 mac_prefix:d2:f0:9      2 max_tunid:16711680 northd_internal_version:22.06.1-20.23.0-63.4 northd_probe_interval:10000 svc_monitor_mac:2a:d8:51:4f:00:e2 use_logical_dp_groups:true]}] Rows:[] Columns:[] Mutati      ons:[] Timeout:<nil> Where:[where column _uuid == {c9788c58-83f6-4f50-a779-23a8d9e7dd8a}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}]
      16296 I0823 02:56:49.902184       1 client.go:781]  "msg"="transacting operations"  "database"="OVN_Northbound" "operations"="[{Op:update Table:NB_Global Row:map[options:{GoMap:map[controll      er_event:true e2e_timestamp:1661223409 mac_prefix:d2:f0:92 max_tunid:16711680 northd_internal_version:22.06.1-20.23.0-63.4 northd_probe_interval:10000 svc_monitor_mac:2a:d8:51:4f:00:e      2 use_logical_dp_groups:true]}] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {c9788c58-83f6-4f50-a779-23a8d9e7dd8a}] Until: Durable:<nil> Comment:<nil> L      ock:<nil> UUIDName:}]"
      16297 W0823 02:56:51.353724       1 obj_retry.go:1031] Dropping retry entry for *v1.CloudPrivateIPConfig 192.168.1.21: exceeded number of failed attempts 

      Expected results:

      The egressIP should be updated correctly.

      Additional info:

            pdiak@redhat.com Patryk Diak
            huirwang Huiran Wang
            Huiran Wang Huiran Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: