Description of problem: FFU from 16.2.5 to 17.1 failed at step 5 of `openstack overcloud upgrade run --yes --stack <stack> --debug --limit allovercloud,undercloud --playbook all` because neutron api was down.

Looking at the container and neutron servers, it was crash because it was unable to load 2 extensions. (network_availability_zone and router_availability_zone).

Looking at the code, I could see that they both required the extension `availability_zone`.

Futherer up the logs we can see what look like neutron not loading the extenision.
~~~
2023-10-06 01:06:22.981 7 DEBUG neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
2023-10-06 01:06:22.981 7 INFO neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Extension availability_zone not supported by any of loaded plugins
~~~

From what I can tell from the code all 3 should be support from the ovn core extensions

Version-Release number of selected component (if applicable):

registry.redhat.io/rhosp-rhel9/openstack-neutron-server:17.1.1-3

How reproducible:

Unsure have only reproduced during FFU so far

Steps to Reproduce:
1. FFU from 16.2.5 with octaiva
(Not sure it necessary with Octavia, it failed with the deploy steps Octavia because the neutron api was down)

Actual results:

Neutron api are down because the neutron container is crashing

Expected results:

Neutron APIs to be up

Additional info:

1. LOGS
   egrep -r "(router_availability_zone|network_availability_zone)" /var/log/containers/neutron/server.log
   2023-10-06 01:01:31.018 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loading extension file: network_availability_zone.py _load_all_extensions_from_path /usr/lib/python3.9/site-packages/neutron/api/extensions.py:450
   2023-10-06 01:01:31.018 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
   2023-10-06 01:01:31.019 8 INFO neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loaded extension: network_availability_zone
   2023-10-06 01:01:31.057 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loading extension file: router_availability_zone.py _load_all_extensions_from_path /usr/lib/python3.9/site-packages/neutron/api/extensions.py:450
   2023-10-06 01:01:31.057 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
   2023-10-06 01:01:31.058 8 INFO neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loaded extension: router_availability_zone
   2023-10-06 01:01:31.105 8 ERROR neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Unable to process extensions (router_availability_zone, network_availability_zone) because the configured plugins do not satisfy their requirements. Some features will not work as expected.
   2023-10-06 01:01:31.105 8 ERROR neutron.service [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Unrecoverable error: please check log for details.: neutron_lib.exceptions.ExtensionsNotFound: Extensions not found: ['router_availability_zone', 'network_availability_zone'].

2023-10-06 01:06:22.981 7 DEBUG neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
2023-10-06 01:06:22.981 7 INFO neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Extension availability_zone not supported by any of loaded plugins

1. ML2 config

[ml2]
type_drivers=geneve,vxlan,vlan,flat
tenant_network_types=geneve,vxlan,vlan,flat
mechanism_drivers=ovn
path_mtu=0
extension_drivers=qos,port_security,dns
overlay_ip_version=4

[ml2_type_geneve]
max_header_size=38
vni_ranges=1:65536

[ml2_type_vxlan]
vxlan_group=224.0.0.1
vni_ranges=1:65536

[ml2_type_vlan]
network_vlan_ranges=tenant:1000:2000

[ml2_type_flat]
flat_networks=datacentre

[ovn]
ovn_nb_connection=ssl:172.17.1.89:6641,ssl:172.17.1.25:6641,ssl:172.17.1.34:6641
ovn_sb_connection=ssl:172.17.1.89:6642,ssl:172.17.1.25:6642,ssl:172.17.1.34:6642
ovn_nb_private_key=/etc/pki/tls/private/neutron_ovn.key
ovn_nb_certificate=/etc/pki/tls/certs/neutron_ovn.crt
ovn_nb_ca_cert=/etc/ipa/ca.crt
ovn_sb_private_key=/etc/pki/tls/private/neutron_ovn.key
ovn_sb_certificate=/etc/pki/tls/certs/neutron_ovn.crt
ovn_sb_ca_cert=/etc/ipa/ca.crt
ovsdb_connection_timeout=180
ovsdb_probe_interval=60000
neutron_sync_mode=log
ovn_metadata_enabled=True
enable_distributed_floating_ip=False
dns_servers=
ovn_emit_need_to_frag=False

[network_log]
rate_limit=100
burst_limit=25

1. Code
   /lib/python3.9/site-packages/neutron/extensions/network_availability_zone.py:

from neutron_lib.api.definitions import availability_zone as az_def
from neutron_lib.api.definitions import network

ALIAS = 'network_availability_zone'
IS_SHIM_EXTENSION = False
IS_STANDARD_ATTR_EXTENSION = False
NAME = 'Network Availability Zone'
API_PREFIX = ''
DESCRIPTION = 'Availability zone support for network.'
UPDATED_TIMESTAMP = '2015-01-01T10:00:00-00:00'
RESOURCE_NAME = network.RESOURCE_NAME
COLLECTION_NAME = network.COLLECTION_NAME
RESOURCE_ATTRIBUTE_MAP = {
COLLECTION_NAME: {
az_def.COLLECTION_NAME:

{ 'allow_post': False, 'allow_put': False, 'is_visible': True }

,
az_def.AZ_HINTS: {
'allow_post': True, 'allow_put': False, 'is_visible': True,
'is_sort_key': True,
'validate':

{'type:availability_zone_hint_list': None}

,
'default': []
}
}
}
SUB_RESOURCE_ATTRIBUTE_MAP = {}
ACTION_MAP = {}
REQUIRED_EXTENSIONS = [az_def.ALIAS]
OPTIONAL_EXTENSIONS = []
ACTION_STATUS = {}

/lib/python3.9/site-packages/neutron/extensions/router_availability_zone.py

from neutron_lib.api.definitions import availability_zone as az
from neutron_lib.api.definitions import l3

ALIAS = 'router_availability_zone'
IS_SHIM_EXTENSION = False
IS_STANDARD_ATTR_EXTENSION = False
NAME = 'Router Availability Zone'
API_PREFIX = ''
DESCRIPTION = 'Availability zone support for router.'
UPDATED_TIMESTAMP = '2015-01-01T10:00:00-00:00'
RESOURCE_NAME = l3.ROUTER
COLLECTION_NAME = l3.ROUTERS
RESOURCE_ATTRIBUTE_MAP = {
COLLECTION_NAME: {
az.COLLECTION_NAME:

{ 'allow_post': False, 'allow_put': False, 'is_visible': True }

,
az.AZ_HINTS: {
'allow_post': True, 'allow_put': False, 'is_visible': True,
'validate':

{'type:availability_zone_hint_list': None}

,
'default': []
}
}
}
SUB_RESOURCE_ATTRIBUTE_MAP = {}
ACTION_MAP = {}
REQUIRED_EXTENSIONS = [l3.ALIAS, az.ALIAS]
OPTIONAL_EXTENSIONS = []
ACTION_STATUS = {}

/lib/python3.9/site-packages/neutron/extensions/availability_zone.py

import abc

from neutron_lib.api.definitions import availability_zone as az_def
from neutron_lib.api import extensions as api_extensions
from neutron_lib.plugins import directory

from neutron.api import extensions
from neutron.api.v2 import base

class Availability_zone(api_extensions.APIExtensionDescriptor):
"""Availability zone extension."""
api_definition = az_def

@classmethod
def get_resources(cls):
"""Returns Ext Resources."""
plugin = directory.get_plugin()
params = az_def.RESOURCE_ATTRIBUTE_MAP.get(az_def.COLLECTION_NAME)
controller = base.create_resource(az_def.COLLECTION_NAME,
az_def.RESOURCE_NAME, plugin, params)

ex = extensions.ResourceExtension(az_def.COLLECTION_NAME, controller)

return [ex]

class AvailabilityZonePluginBase(object, metaclass=abc.ABCMeta):
"""REST API to operate the Availability Zone."""

@abc.abstractmethod
def get_availability_zones(self, context, filters=None, fields=None,
sorts=None, limit=None, marker=None,
page_reverse=False):
"""Return availability zones which a resource belongs to"""

@abc.abstractmethod
def validate_availability_zones(self, context, resource_type,
availability_zones):
"""Verify that the availability zones exist."""

/lib/python3.9/site-packages/neutron/common/ovn/extensions.py

from neutron_lib.api.definitions import address_scope
from neutron_lib.api.definitions import agent as agent_def
from neutron_lib.api.definitions import allowedaddresspairs
from neutron_lib.api.definitions import auto_allocated_topology
from neutron_lib.api.definitions import availability_zone as az_def <---------------- Availability zone extension
from neutron_lib.api.definitions import default_subnetpools
from neutron_lib.api.definitions import dns
from neutron_lib.api.definitions import dns_domain_ports
from neutron_lib.api.definitions import expose_port_forwarding_in_fip
from neutron_lib.api.definitions import external_net
from neutron_lib.api.definitions import extra_dhcp_opt
from neutron_lib.api.definitions import extraroute
from neutron_lib.api.definitions import filter_validation
from neutron_lib.api.definitions import fip_pf_description
from neutron_lib.api.definitions import fip_port_details
from neutron_lib.api.definitions import flavors
from neutron_lib.api.definitions import floating_ip_port_forwarding
from neutron_lib.api.definitions import floatingip_pools
from neutron_lib.api.definitions import l3
from neutron_lib.api.definitions import l3_ext_gw_mode
from neutron_lib.api.definitions import l3_flavors
from neutron_lib.api.definitions import logging
from neutron_lib.api.definitions import multiprovidernet
from neutron_lib.api.definitions import network_availability_zone <---------------- Network Availability Zone
from neutron_lib.api.definitions import network_ip_availability
from neutron_lib.api.definitions import network_mtu
from neutron_lib.api.definitions import network_mtu_writable
from neutron_lib.api.definitions import pagination
from neutron_lib.api.definitions import port_device_profile
from neutron_lib.api.definitions import port_mac_address_regenerate
from neutron_lib.api.definitions import port_numa_affinity_policy
from neutron_lib.api.definitions import port_resource_request
from neutron_lib.api.definitions import port_security
from neutron_lib.api.definitions import portbindings
from neutron_lib.api.definitions import portbindings_extended as pbe_ext
from neutron_lib.api.definitions import project_id
from neutron_lib.api.definitions import provider_net
from neutron_lib.api.definitions import qos
from neutron_lib.api.definitions import qos_bw_limit_direction
from neutron_lib.api.definitions import qos_default
from neutron_lib.api.definitions import qos_gateway_ip
from neutron_lib.api.definitions import qos_rule_type_details
from neutron_lib.api.definitions import qos_rules_alias
from neutron_lib.api.definitions import rbac_address_scope
from neutron_lib.api.definitions import rbac_security_groups
from neutron_lib.api.definitions import router_availability_zone as raz_def
from neutron_lib.api.definitions import security_groups_normalized_cidr
from neutron_lib.api.definitions import security_groups_remote_address_group
from neutron_lib.api.definitions import segment as seg_def
from neutron_lib.api.definitions import sorting
from neutron_lib.api.definitions import stateful_security_group
from neutron_lib.api.definitions import subnet_dns_publish_fixed_ip
from neutron_lib.api.definitions import trunk
from neutron_lib.api.definitions import vlantransparent
from neutron_lib import constants

from neutron.extensions import security_groups_shared_filtering_lib

1. NOTE(russellb) This remains in its own file (vs constants.py) because we want
2. to be able to easily import it and export the info without any dependencies
3. on external imports.

1. NOTE(russellb) If you update these lists, please also update
2. doc/source/admin/ovn/features.rst and the current release note.
   ML2_SUPPORTED_API_EXTENSIONS_OVN_L3 = [
   l3.ALIAS,
   extraroute.ALIAS,
   l3_ext_gw_mode.ALIAS,
   fip_port_details.ALIAS,
   floatingip_pools.ALIAS,
   pagination.ALIAS,
   'qos-fip',
   qos_gateway_ip.ALIAS,
   sorting.ALIAS,
   project_id.ALIAS,
   dns.ALIAS,
   dns_domain_ports.ALIAS,
   subnet_dns_publish_fixed_ip.ALIAS,
   agent_def.ALIAS,
   az_def.ALIAS, <---------------- Availability zone extension
   raz_def.ALIAS, <---------------- Router Availability Zone
   flavors.ALIAS,
   l3_flavors.ALIAS,
   ]
   ML2_SUPPORTED_API_EXTENSIONS = [
   address_scope.ALIAS,
   agent_def.ALIAS,
   allowedaddresspairs.ALIAS,
   auto_allocated_topology.ALIAS,
   portbindings.ALIAS,
   pbe_ext.ALIAS,
   default_subnetpools.ALIAS,
   dns.ALIAS,
   external_net.ALIAS,
   extra_dhcp_opt.ALIAS,
   filter_validation.ALIAS,
   multiprovidernet.ALIAS,
   network_mtu.ALIAS,
   network_mtu_writable.ALIAS,
   network_availability_zone.ALIAS, <---------------- Network Availability Zone
   network_ip_availability.ALIAS,
   port_device_profile.ALIAS,
   port_mac_address_regenerate.ALIAS,
   port_numa_affinity_policy.ALIAS,
   port_security.ALIAS,
   provider_net.ALIAS,
   port_resource_request.ALIAS,
   qos.ALIAS,
   qos_bw_limit_direction.ALIAS,
   qos_default.ALIAS,
   qos_rule_type_details.ALIAS,
   qos_rules_alias.ALIAS,
   'quotas',
   rbac_address_scope.ALIAS,
   'rbac-policies',
   rbac_security_groups.ALIAS,
   'standard-attr-revisions',
   'security-group',
   security_groups_normalized_cidr.ALIAS,
   security_groups_remote_address_group.ALIAS,
   stateful_security_group.ALIAS,
   security_groups_shared_filtering_lib.ALIAS,
   'standard-attr-description',
   constants.SUBNET_ALLOCATION_EXT_ALIAS,
   'standard-attr-tag',
   'standard-attr-timestamp',
   trunk.ALIAS,
   'quota_details',
   seg_def.ALIAS,
   expose_port_forwarding_in_fip.ALIAS,
   fip_pf_description.ALIAS,
   floating_ip_port_forwarding.ALIAS,
   vlantransparent.ALIAS,
   logging.ALIAS,
   ]