-
Bug
-
Resolution: Done
-
Critical
-
None
-
None
-
None
-
5
-
False
-
-
False
-
?
-
None
-
-
-
Important
It seems that there are some Neutron database inconsistency during the upgrade.
I think we need to open a Jira ticket to get help from Engineering.
The following error message is complaining that "neutron-db-manage upgrade heads" failed due to "Table 'ovn_revision_numbers' already exists" error.
++ cat /run_command\ + CMD='/usr/bin/bootstrap_host_exec neutron_api neutron-db-manage upgrade heads'\ + ARGS=\ + [[ ! -n '' ]]\ + . kolla_extend_start\ + echo 'Running command: '\\\\''/usr/bin/bootstrap_host_exec neutron_api neutron-db-manage upgrade heads'\\\\'''\ + umask 0022\ + exec /usr/bin/bootstrap_host_exec neutron_api neutron-db-manage upgrade heads\ INFO [alembic.runtime.migration] Context impl MySQLImpl.\ INFO [alembic.runtime.migration] Will assume non-transactional DDL.\ INFO [alembic.runtime.migration] Context impl MySQLImpl.\ INFO [alembic.runtime.migration] Will assume non-transactional DDL.\ INFO [alembic.runtime.migration] Running upgrade 86274d77933e -> f4b9654dd40c\ : sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1050, \\\"Table 'ovn_revision_numbers' already exists\\\")\ [SQL: \ CREATE TABLE ovn_revision_numbers (\ \\tstandard_attr_id BIGINT, \ \\tresource_uuid VARCHAR(36) NOT NULL, \ \\tresource_type VARCHAR(36) NOT NULL, \ \\trevision_number BIGINT NOT NULL DEFAULT '0', \ \\tcreated_at DATETIME NOT NULL, \ \\tupdated_at TIMESTAMP NULL, \ \\tPRIMARY KEY (resource_uuid, resource_type), \ \\tFOREIGN KEY(standard_attr_id) REFERENCES standardattributes (id) ON DELETE SET NULL\ )ENGINE=InnoDB\ \ ]\
"neutron-db-manage upgrade heads" is a command to upgrade the Neutron database schema during OpenStack upgrade.
https://docs.openstack.org/neutron/wallaby/admin/migration-database.html After installing a new version of the Neutron server, upgrade the database using the following command: $ neutron-db-manage upgrade heads
This command internally use Alembic to manage the database schema version.
Ref: https://alembic.sqlalchemy.org/en/latest/cookbook.html#create-revision-migrations
Ref: https://alembic.sqlalchemy.org/en/latest/tutorial.html#create-a-migration-script
Ref: https://alembic.sqlalchemy.org/en/latest/cookbook.html#create-revision-migrations
In Alembic, we have "revision" which identify the revision of the database schema.
OpenStack Neutron database schema upgrade is done by updating the database schema from a old revision to a new revision one-by-one.
It seems that the issue occurred when Neutron DB schema is updated from the old revision "86274d77933e" to a new revision "f4b9654dd40c".
https://github.com/openstack/neutron/blob/unmaintained/wallaby/neutron/db/migration/alembic_migrations/versions/ussuri/expand/f4b9654dd40c_ovn_backend.py # revision identifiers, used by Alembic. revision = 'f4b9654dd40c' down_revision = '86274d77933e' : op.create_table( OVN_REVISION_NUMBERS, sa.Column('standard_attr_id', sa.BigInteger, nullable=True), sa.Column('resource_uuid', sa.String(36), nullable=False, index=True), sa.Column('resource_type', sa.String(36), nullable=False, index=True), sa.Column('revision_number', sa.BigInteger, nullable=False, server_default='0'), sa.Column('created_at', sa.DateTime, nullable=False, default=sa.func.now()), sa.Column('updated_at', sa.TIMESTAMP, default=sa.func.now(), onupdate=sa.func.now(), nullable=True), sa.ForeignKeyConstraint( ['standard_attr_id'], ['standardattributes.id'], ondelete='SET NULL'), sa.PrimaryKeyConstraint('resource_uuid', 'resource_type') )
When Neutron DB schema is updated from the old revision "86274d77933e" to a new revision "f4b9654dd40c", ovn_revision_numbers table is supposed to be newly created.
However, in the customer's environment, the ovn_revision_numbers table exist already.
That's why the issue occurs.
Therefore, the Neutron DB has some inconsistency between its revision and actual database status.
I suppose we need to identify its actual revision from the database dump and solve the inconsistency by making some changes in the Neutron DB.
