We noticed that scenario test tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize is failing randomly in downstream CI. Examples of the failure:

http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-neutron-16.2_director-rhel-virthost-3cont_2comp-ml2ovs-ipv6-all-vxlan-tempest/49/test_results/tempest-results-neutron.1.html where failure was like below:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tempest/common/utils/_init_.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_advanced_server_ops.py", line 196, in test_server_connectivity_resize
'VERIFY_RESIZE')
File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 78, in wait_for_server_status
raise exceptions.BuildErrorException(server_id=server_id)
tempest.exceptions.BuildErrorException: Server edba0ed0-d6e1-4b1d-bc35-3a8d89d397f0 failed to build and is in ERROR status

Unfortunately there are no logs other than test results from this run.

But there are other 2 examples of similar failures of that test:

• http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-neutron-16.2_director-rhel-virthost-3cont_2comp-ml2ovs-ipv6-all-vxlan-tempest/54/test_results/tempest-results-neutron.1.html
• http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-neutron-16.2_director-rhel-virthost-3cont_2comp-ml2ovs-ipv6-all-vxlan-tempest/46/test_results/tempest-results-neutron.1.html

where failure was like below:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tempest/common/utils/_init_.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_advanced_server_ops.py", line 196, in test_server_connectivity_resize
'VERIFY_RESIZE')
File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (TestNetworkAdvancedServerOps:test_server_connectivity_resize) Server 9460eca7-30ac-4ffe-9ef7-aa7e9b241f3b failed to reach VERIFY_RESIZE status and task state "None" within the required time (300 s). Current status: ACTIVE. Current task state: None.

After checking nova-compute logs from of of those runs (http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-neutron-16.2_director-rhel-virthost-3cont_2comp-ml2ovs-ipv6-all-vxlan-tempest/54/compute-0/var/log/containers/nova/nova-compute.log.gz) I found out some potentially related deadlock there. It is in http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-neutron-16.2_director-rhel-virthost-3cont_2comp-ml2ovs-ipv6-all-vxlan-tempest/54/compute-0/var/log/containers/nova/nova-compute.log.gz and looks like below:

2024-03-13 22:15:12.943 7 INFO nova.compute.manager [req-184848ab-aa09-4c01-8b19-d44cb5c5e0a1 0e2b931589ca40d4ba7c9d22e3678513 18a715f33c0e4b79bdc8b3265d3ae94a - default default] [instance: 0cf6061f-007f-4b31-afa2-3f4e9b5a739f] Successfully reverted task state from resize_prep on failure for instance.
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server [req-184848ab-aa09-4c01-8b19-d44cb5c5e0a1 0e2b931589ca40d4ba7c9d22e3678513 18a715f33c0e4b79bdc8b3265d3ae94a - default default] Exception during message handling: oslo_messaging.rpc.client.RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1213, 'Deadlock: wsrep aborted transaction')
[SQL: UPDATE instance_actions SET updated_at=%(updated_at)s WHERE instance_actions.id = %(instance_actions_id)s]
[parameters:

{'updated_at': datetime.datetime(2024, 3, 13, 22, 15, 12, 801841), 'instance_actions_id': 426}

]
(Background on this error at: http://sqlalche.me/e/e3q8)
['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in execute_context\n cursor, statement, parameters, context\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', "pymysql.err.OperationalError: (1213, 'Deadlock: wsrep aborted transaction')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 135, in _object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper\n result = fn(cls, context, *args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/objects/instance_action.py", line 201, in event_start\n db_event = db.action_event_start(context, values)\n', ' File "/usr/lib/python3.6/site-packages/nova/db/api.py", line 1691, in action_event_start\n return IMPL.action_event_start(context, values)\n', ' File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit\n self.force_reraise()\n', ' File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise\n raise value\n', ' File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 224, in wrapped\n return f(context, *args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 5290, in action_event_start\n action.save(context.session)\n', ' File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save\n session.flush()\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2451, in flush\n self.flush(objects)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2589, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit_\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise\n raise value\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2549, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute\n uow,\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj\n update,\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 978, in _emit_update_statements\n statement, multiparams\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute\n return meth(self, multiparams, params)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement\n distilled_params,\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context\n e, statement, parameters, cursor, context\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception\n util.raise_from_cause(newraise, exc_info)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise\n raise value.with_traceback(tb)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context\n cursor, statement, parameters, context\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', "sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1213, 'Deadlock: wsrep aborted transaction')\n[SQL: UPDATE instance_actions SET updated_at=%(updated_at)s WHERE instance_actions.id = %(instance_actions_id)s]\n[parameters:

{'updated_at': datetime.datetime(2024, 3, 13, 22, 15, 12, 801841), 'instance_actions_id': 426}

]\n(Background on this error at: http://sqlalche.me/e/e3q8)\n"].
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in _exit_
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server raise value
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 191, in decorated_function
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in _exit_
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server raise value
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 161, in decorated_function
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1386, in decorated_function
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server with EventReporter(context, event_name, host, instance_uuid):
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1356, in _enter_
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server host=self.host)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server args, kwargs)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server args=args, kwargs=kwargs)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server transport_options=self.transport_options)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server transport_options=transport_options)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 674, in send
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server transport_options=transport_options)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 664, in _send
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server raise result
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server oslo_messaging.rpc.client.RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1213, 'Deadlock: wsrep aborted transaction')
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server [SQL: UPDATE instance_actions SET updated_at=%(updated_at)s WHERE instance_actions.id = %(instance_actions_id)s]
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server [parameters:

{'updated_at': datetime.datetime(2024, 3, 13, 22, 15, 12, 801841), 'instance_actions_id': 426}

]
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server (Background on this error at: http://sqlalche.me/e/e3q8)
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in execute_context\n cursor, statement, parameters, context\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', "pymysql.err.OperationalError: (1213, 'Deadlock: wsrep aborted transaction')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 135, in _object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper\n result = fn(cls, context, *args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/objects/instance_action.py", line 201, in event_start\n db_event = db.action_event_start(context, values)\n', ' File "/usr/lib/python3.6/site-packages/nova/db/api.py", line 1691, in action_event_start\n return IMPL.action_event_start(context, values)\n', ' File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit\n self.force_reraise()\n', ' File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise\n raise value\n', ' File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 224, in wrapped\n return f(context, *args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 5290, in action_event_start\n action.save(context.session)\n', ' File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save\n session.flush()\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2451, in flush\n self.flush(objects)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2589, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit_\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise\n raise value\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2549, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute\n uow,\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj\n update,\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 978, in _emit_update_statements\n statement, multiparams\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute\n return meth(self, multiparams, params)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement\n distilled_params,\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context\n e, statement, parameters, cursor, context\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception\n util.raise_from_cause(newraise, exc_info)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise\n raise value.with_traceback(tb)\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context\n cursor, statement, parameters, context\n', ' File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', "sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1213, 'Deadlock: wsrep aborted transaction')\n[SQL: UPDATE instance_actions SET updated_at=%(updated_at)s WHERE instance_actions.id = %(instance_actions_id)s]\n[parameters:

{'updated_at': datetime.datetime(2024, 3, 13, 22, 15, 12, 801841), 'instance_actions_id': 426}

]\n(Background on this error at: http://sqlalche.me/e/e3q8)\n"].
2024-03-13 22:15:12.950 7 ERROR oslo_messaging.rpc.server