Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-208

Race condition when creating / deleting mac_address_pairs

    XMLWordPrintable

Details

    • SDN Sprint 223, SDN Sprint 224, SDN Sprint 225
    • 3
    • False
    • Hide

      None

      Show
      None

    Description

      We currently use the version number approach in neutron to avoid race conditions when updating allowed_address_pairs. However, this is clearly not working because of the following neutron bug: https://bugzilla.redhat.com/show_bug.cgi?id=2119199

      [stack@standalone egressip]$ for i in {100..110}; do oc delete -f egressip$i.yaml; done
      egressip.k8s.ovn.org "egressip100" deleted
      egressip.k8s.ovn.org "egressip101" deleted
      Error from server (NotFound): error when deleting "egressip102.yaml": egressips.k8s.ovn.org "egressip102" not found
      egressip.k8s.ovn.org "egressip103" deleted
      egressip.k8s.ovn.org "egressip104" deleted
      egressip.k8s.ovn.org "egressip105" deleted
      egressip.k8s.ovn.org "egressip106" deleted
      egressip.k8s.ovn.org "egressip107" deleted
      egressip.k8s.ovn.org "egressip108" deleted
      egressip.k8s.ovn.org "egressip109" deleted
      egressip.k8s.ovn.org "egressip110" deleted
      [stack@standalone egressip]$ ll
      total 56
      -rw-rw-r--. 1 stack stack 365 Aug 17 15:24 debug.yaml
      -rw-rw-r--. 1 stack stack 520 Jul 19 13:09 deploy.sh
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:27 egressip100.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:27 egressip101.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:29 egressip102.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip103.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip104.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip105.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip106.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip107.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip108.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip109.yaml
      -rw-rw-r--. 1 stack stack 191 Aug 17 15:32 egressip110.yaml
      -rw-rw-r--. 1 stack stack 641 Jul 14 13:35 fedora.yaml
      [stack@standalone egressip]$ oc get egressip
      NAME        EGRESSIPS       ASSIGNED NODE                 ASSIGNED EGRESSIPS
      egressip2   10.196.10.102   ostest-lq27s-worker-0-cz6gw   10.196.10.102
      [stack@standalone egressip]$ openstack port show ostest-lq27s-worker-0-cz6gw-0
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | Field                   | Value                                                                                                                                                                                    |
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | admin_state_up          | UP                                                                                                                                                                                       |
      | allowed_address_pairs   | ip_address='10.196.0.5', mac_address='fa:16:3e:fe:80:35'                                                                                                                                 |
      |                         | ip_address='10.196.0.7', mac_address='fa:16:3e:fe:80:35'                                                                                                                                 |
      |                         | ip_address='10.196.10.100', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.102', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.103', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.104', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      | binding_host_id         | None                                                                                                                                                                                     |
      | binding_profile         | None                                                                                                                                                                                     |
      | binding_vif_details     | None                                                                                                                                                                                     |
      | binding_vif_type        | None                                                                                                                                                                                     |
      | binding_vnic_type       | normal                                                                                                                                                                                   |
      | created_at              | 2022-07-14T00:28:00Z                                                                                                                                                                     |
      | data_plane_status       | None                                                                                                                                                                                     |
      | description             | Created by cluster-api-provider-openstack cluster openshift-machine-api-ostest-lq27s                                                                                                     |
      | device_id               | 382c107f-a082-4e9b-8adb-2ba45323c479                                                                                                                                                     |
      | device_owner            | compute:nova                                                                                                                                                                             |
      | dns_assignment          | fqdn='ostest-lq27s-worker-0-cz6gw.shiftstack.', hostname='ostest-lq27s-worker-0-cz6gw', ip_address='10.196.2.215'                                                                        |
      | dns_domain              | None                                                                                                                                                                                     |
      | dns_name                | ostest-lq27s-worker-0-cz6gw                                                                                                                                                              |
      | extra_dhcp_opts         |                                                                                                                                                                                          |
      | fixed_ips               | ip_address='10.196.2.215', subnet_id='ba4e5cdb-a0e3-47f2-9233-47d5a444412c'                                                                                                              |
      | id                      | 2ec4488a-35c2-4a14-adfb-a9ec3af61154                                                                                                                                                     |
      | location                | cloud='standalone_openshift', project.domain_id=, project.domain_name='Default', project.id='6f69c87fe95b469c8c7709382818acf6', project.name='openshift', region_name='regionOne', zone= |
      | mac_address             | fa:16:3e:fe:80:35                                                                                                                                                                        |
      | name                    | ostest-lq27s-worker-0-cz6gw-0                                                                                                                                                            |
      | network_id              | ed889e25-f8fa-4684-a9c4-54fff8de37b8                                                                                                                                                     |
      | port_security_enabled   | True                                                                                                                                                                                     |
      | project_id              | 6f69c87fe95b469c8c7709382818acf6                                                                                                                                                         |
      | propagate_uplink_status | None                                                                                                                                                                                     |
      | qos_policy_id           | None                                                                                                                                                                                     |
      | resource_request        | None                                                                                                                                                                                     |
      | revision_number         | 32                                                                                                                                                                                       |
      | security_group_ids      | d426fa6b-74e1-4272-95c0-e7edd4978cef                                                                                                                                                     |
      | status                  | ACTIVE                                                                                                                                                                                   |
      | tags                    | cluster-api-provider-openstack, openshift-machine-api-ostest-lq27s, openshiftClusterID=ostest-lq27s                                                                                      |
      | trunk_details           | {'trunk_id': '5e200918-6cc9-46e6-a444-67b19a02ff3a', 'sub_ports': []}                                                                                                                    |
      | updated_at              | 2022-08-17T15:36:10Z                                                                                                                                                                     |
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      [stack@standalone egressip]$ 
      [stack@standalone egressip]$ 
      [stack@standalone egressip]$ 
      [stack@standalone egressip]$ 
      [stack@standalone egressip]$ 
      [stack@standalone egressip]$ 
      [stack@standalone egressip]$ 
      [stack@standalone egressip]$ openstack port show ostest-lq27s-worker-0-cz6gw-0
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | Field                   | Value                                                                                                                                                                                    |
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | admin_state_up          | UP                                                                                                                                                                                       |
      | allowed_address_pairs   | ip_address='10.196.0.5', mac_address='fa:16:3e:fe:80:35'                                                                                                                                 |
      |                         | ip_address='10.196.0.7', mac_address='fa:16:3e:fe:80:35'                                                                                                                                 |
      |                         | ip_address='10.196.10.100', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.102', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.103', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.104', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      | binding_host_id         | None                                                                                                                                                                                     |
      | binding_profile         | None                                                                                                                                                                                     |
      | binding_vif_details     | None                                                                                                                                                                                     |
      | binding_vif_type        | None                                                                                                                                                                                     |
      | binding_vnic_type       | normal                                                                                                                                                                                   |
      | created_at              | 2022-07-14T00:28:00Z                                                                                                                                                                     |
      | data_plane_status       | None                                                                                                                                                                                     |
      | description             | Created by cluster-api-provider-openstack cluster openshift-machine-api-ostest-lq27s                                                                                                     |
      | device_id               | 382c107f-a082-4e9b-8adb-2ba45323c479                                                                                                                                                     |
      | device_owner            | compute:nova                                                                                                                                                                             |
      | dns_assignment          | fqdn='ostest-lq27s-worker-0-cz6gw.shiftstack.', hostname='ostest-lq27s-worker-0-cz6gw', ip_address='10.196.2.215'                                                                        |
      | dns_domain              | None                                                                                                                                                                                     |
      | dns_name                | ostest-lq27s-worker-0-cz6gw                                                                                                                                                              |
      | extra_dhcp_opts         |                                                                                                                                                                                          |
      | fixed_ips               | ip_address='10.196.2.215', subnet_id='ba4e5cdb-a0e3-47f2-9233-47d5a444412c'                                                                                                              |
      | id                      | 2ec4488a-35c2-4a14-adfb-a9ec3af61154                                                                                                                                                     |
      | location                | cloud='standalone_openshift', project.domain_id=, project.domain_name='Default', project.id='6f69c87fe95b469c8c7709382818acf6', project.name='openshift', region_name='regionOne', zone= |
      | mac_address             | fa:16:3e:fe:80:35                                                                                                                                                                        |
      | name                    | ostest-lq27s-worker-0-cz6gw-0                                                                                                                                                            |
      | network_id              | ed889e25-f8fa-4684-a9c4-54fff8de37b8                                                                                                                                                     |
      | port_security_enabled   | True                                                                                                                                                                                     |
      | project_id              | 6f69c87fe95b469c8c7709382818acf6                                                                                                                                                         |
      | propagate_uplink_status | None                                                                                                                                                                                     |
      | qos_policy_id           | None                                                                                                                                                                                     |
      | resource_request        | None                                                                                                                                                                                     |
      | revision_number         | 32                                                                                                                                                                                       |
      | security_group_ids      | d426fa6b-74e1-4272-95c0-e7edd4978cef                                                                                                                                                     |
      | status                  | ACTIVE                                                                                                                                                                                   |
      | tags                    | cluster-api-provider-openstack, openshift-machine-api-ostest-lq27s, openshiftClusterID=ostest-lq27s                                                                                      |
      | trunk_details           | {'trunk_id': '5e200918-6cc9-46e6-a444-67b19a02ff3a', 'sub_ports': []}                                                                                                                    |
      | updated_at              | 2022-08-17T15:36:10Z                                                                                                                                                                     |
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      [stack@standalone egressip]$ openshift get pods ^C
      [stack@standalone egressip]$ openstack port list | grep 10.196.10.103
      [stack@standalone egressip]$ openstack port show ostest-lq27s-worker-0-cz6gw-0
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | Field                   | Value                                                                                                                                                                                    |
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | admin_state_up          | UP                                                                                                                                                                                       |
      | allowed_address_pairs   | ip_address='10.196.0.5', mac_address='fa:16:3e:fe:80:35'                                                                                                                                 |
      |                         | ip_address='10.196.0.7', mac_address='fa:16:3e:fe:80:35'                                                                                                                                 |
      |                         | ip_address='10.196.10.100', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.102', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.103', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      |                         | ip_address='10.196.10.104', mac_address='fa:16:3e:fe:80:35'                                                                                                                              |
      | binding_host_id         | None                                                                                                                                                                                     |
      | binding_profile         | None                                                                                                                                                                                     |
      | binding_vif_details     | None                                                                                                                                                                                     |
      | binding_vif_type        | None                                                                                                                                                                                     |
      | binding_vnic_type       | normal                                                                                                                                                                                   |
      | created_at              | 2022-07-14T00:28:00Z                                                                                                                                                                     |
      | data_plane_status       | None                                                                                                                                                                                     |
      | description             | Created by cluster-api-provider-openstack cluster openshift-machine-api-ostest-lq27s                                                                                                     |
      | device_id               | 382c107f-a082-4e9b-8adb-2ba45323c479                                                                                                                                                     |
      | device_owner            | compute:nova                                                                                                                                                                             |
      | dns_assignment          | fqdn='ostest-lq27s-worker-0-cz6gw.shiftstack.', hostname='ostest-lq27s-worker-0-cz6gw', ip_address='10.196.2.215'                                                                        |
      | dns_domain              | None                                                                                                                                                                                     |
      | dns_name                | ostest-lq27s-worker-0-cz6gw                                                                                                                                                              |
      | extra_dhcp_opts         |                                                                                                                                                                                          |
      | fixed_ips               | ip_address='10.196.2.215', subnet_id='ba4e5cdb-a0e3-47f2-9233-47d5a444412c'                                                                                                              |
      | id                      | 2ec4488a-35c2-4a14-adfb-a9ec3af61154                                                                                                                                                     |
      | location                | cloud='standalone_openshift', project.domain_id=, project.domain_name='Default', project.id='6f69c87fe95b469c8c7709382818acf6', project.name='openshift', region_name='regionOne', zone= |
      | mac_address             | fa:16:3e:fe:80:35                                                                                                                                                                        |
      | name                    | ostest-lq27s-worker-0-cz6gw-0                                                                                                                                                            |
      | network_id              | ed889e25-f8fa-4684-a9c4-54fff8de37b8                                                                                                                                                     |
      | port_security_enabled   | True                                                                                                                                                                                     |
      | project_id              | 6f69c87fe95b469c8c7709382818acf6                                                                                                                                                         |
      | propagate_uplink_status | None                                                                                                                                                                                     |
      | qos_policy_id           | None                                                                                                                                                                                     |
      | resource_request        | None                                                                                                                                                                                     |
      | revision_number         | 32                                                                                                                                                                                       |
      | security_group_ids      | d426fa6b-74e1-4272-95c0-e7edd4978cef                                                                                                                                                     |
      | status                  | ACTIVE                                                                                                                                                                                   |
      | tags                    | cluster-api-provider-openstack, openshift-machine-api-ostest-lq27s, openshiftClusterID=ostest-lq27s                                                                                      |
      | trunk_details           | {'trunk_id': '5e200918-6cc9-46e6-a444-67b19a02ff3a', 'sub_ports': []}                                                                                                                    |
      | updated_at              | 2022-08-17T15:36:10Z                                                                                                                                                                     |
      +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      [stack@standalone egressip]$ 
      

      Attachments

        Activity

          People

            akaris@redhat.com Andreas Karis
            akaris@redhat.com Andreas Karis
            Huiran Wang Huiran Wang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: