Description of problem:
======================

We have an OSP 16.2.6 case where customer is unable to launch a VM with SR-IOV.
The error seen is:
Timeout waiting for [('network-vif-plugged', 'f4d628c8-fa66-4be9-a072-7315081eac98')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds

Looking at timeline below, it looks as if the Neutron PUT command [2] gets sent before Nova is ready for it at [3].

• • • Looking at previous logs, since new logs don't have DEBUG enabled on controllers for Neutron

[1] compute 25 "old" nova logs: ./0020-devin_sosreport_2024-08-27.tar.gz/devin_sosreport_2024-08-27/sosreport-maynard3-compute025-2024-08-27-ytrzcmw.tar.xz/sosreport-maynard3-compute025-2024-08-27-ytrzcmw/var/log/containers/nova

1. 1. 1. Starting instance...
         nova-compute.log:2024-08-27 17:31:40.443 8 DEBUG nova.compute.manager [req-059f7297-3764-43dc-9d03-f2fa6f9ec34f ab21a1d7b3e1d8359a8f54d3fd4c9f6e819539e742c0dea5cc41116ff252fd49 cb1b3085a4ca4b7fb6bedf40db34edcc - 5a47fcfd1bbd478ead6ca50957811fca 5a47fcfd1bbd478ead6ca50957811fca] [instance: 4960ad8c-6206-4ac7-b81d-f72dace88926] Starting instance... _do_build_and_run_instance /usr/lib/python3.6/site-packages/nova/compute/manager.py:2241

[2] controller 3 "old" neutron logs: ./0020-devin_sosreport_2024-08-27.tar.gz/devin_sosreport_2024-08-27/sosreport-maynard3-controller003-2024-08-27-soqqzyp.tar.xz/sosreport-maynard3-controller003-2024-08-27-soqqzyp/var/log/containers/neutron

1. 1. 1. Attempting to bind port f4d628c8-fa66-4be9-a072-7315081eac98 on host maynard3-compute025.localdomain ...
         server.log:2024-08-27 17:31:42.450 23 DEBUG neutron.plugins.ml2.managers [req-0cfd7fa2-46d6-44dd-a2e6-57e2f1af1648 de567cbc7f2b4ef39d4f93d583c772ca 403b4fe78586414da1654e052c44e69a - default default] Attempting to bind port f4d628c8-fa66-4be9-a072-7315081eac98 on host maynard3-compute025.localdomain for vnic_type direct with profile {"pci_vendor_info": "8086:154c", "pci_slot": "0000:d8:04.3", "physical_network": "sriov-1-1"}

         bind_port /usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py:795

1. 1. 1. PUT /v2.0/ports/f4d628c8-fa66-4be9-a072-7315081eac98 :
         server.log:2024-08-27 17:31:42.549 23 INFO neutron.wsgi [req-0cfd7fa2-46d6-44dd-a2e6-57e2f1af1648 de567cbc7f2b4ef39d4f93d583c772ca 403b4fe78586414da1654e052c44e69a - default default] 192.168.143.136,192.168.143.89 "PUT /v2.0/ports/f4d628c8-fa66-4be9-a072-7315081eac98 HTTP/1.1" status: 200 len: 1308 time: 0.6444976

[3] compute 25 "old" nova logs: ./0020-devin_sosreport_2024-08-27.tar.gz/devin_sosreport_2024-08-27/sosreport-maynard3-compute025-2024-08-27-ytrzcmw.tar.xz/sosreport-maynard3-compute025-2024-08-27-ytrzcmw/var/log/containers/nova

1. 1. 1. Preparing to wait for external event network-vif-plugged-f4d628c8-fa66-4be9-a072-7315081eac98 :
         nova-compute.log:2024-08-27 17:35:39.264 8 DEBUG nova.compute.manager [req-059f7297-3764-43dc-9d03-f2fa6f9ec34f ab21a1d7b3e1d8359a8f54d3fd4c9f6e819539e742c0dea5cc41116ff252fd49 cb1b3085a4ca4b7fb6bedf40db34edcc - 5a47fcfd1bbd478ead6ca50957811fca 5a47fcfd1bbd478ead6ca50957811fca] [instance: 4960ad8c-6206-4ac7-b81d-f72dace88926] Preparing to wait for external event network-vif-plugged-f4d628c8-fa66-4be9-a072-7315081eac98 prepare_for_instance_event /usr/lib/python3.6/site-packages/nova/compute/manager.py:323

[4] compute 25 "old" nova logs: ./0020-devin_sosreport_2024-08-27.tar.gz/devin_sosreport_2024-08-27/sosreport-maynard3-compute025-2024-08-27-ytrzcmw.tar.xz/sosreport-maynard3-compute025-2024-08-27-ytrzcmw/var/log/containers/nova

1. 1. 1. Timeout waiting for 'network-vif-plugged' after 300 seconds :
         nova-compute.log:2024-08-27 17:40:46.266 8 WARNING nova.virt.libvirt.driver [req-059f7297-3764-43dc-9d03-f2fa6f9ec34f ab21a1d7b3e1d8359a8f54d3fd4c9f6e819539e742c0dea5cc41116ff252fd49 cb1b3085a4ca4b7fb6bedf40db34edcc - 5a47fcfd1bbd478ead6ca50957811fca 5a47fcfd1bbd478ead6ca50957811fca] [instance: 4960ad8c-6206-4ac7-b81d-f72dace88926] Timeout waiting for [('network-vif-plugged', 'f4d628c8-fa66-4be9-a072-7315081eac98')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds

[5] controller 2 "old" neutron logs: ./0020-devin_sosreport_2024-08-27.tar.gz/devin_sosreport_2024-08-27/sosreport-maynard3-controller002-2024-08-27-bjcpqui.tar.xz/sosreport-maynard3-controller002-2024-08-27-bjcpqui/var/log/containers/neutron

1. 1. 1. This command likely tied to nova timing out (cleanup, etc)
         server.log:2024-08-27 17:40:48.518 24 INFO neutron.wsgi [req-7bd16ce3-eb3b-4a44-9460-d35499725ab7 de567cbc7f2b4ef39d4f93d583c772ca 403b4fe78586414da1654e052c44e69a - default default] 192.168.143.136,192.168.143.89 "PUT /v2.0/ports/f4d628c8-fa66-4be9-a072-7315081eac98 HTTP/1.1" status: 200 len: 1090 time: 0.4029467

Version-Release number of selected component (if applicable):
============================================================
RHOSP 16.2.6

How reproducible:
================
Always

Steps to Reproduce:
==================
On customer system, launch instance with SR-IOV interface.
Target computer is empty (no instances running on it).

Actual results:
==============
Timeout error, instance cannot be created

Expected results:
================
Instance gets created

Additional info:
===============