ERROR ironic.conductor.base_manager sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'ironic.conductors' doesn't exist")

Version:
4.10.18

The cluster type is non-platform. The cluster was deployed on RH openstack.

Deployed multicluster-engine.v2.0.0 on the cluster

The metal3 was enabled using the following:

apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
name: provisioning-configuration
spec:
provisioningNetwork: "Disabled"
watchAllNamespaces: true

If the metal3 pod is deleted - then after restart looking at the metal3-ironic-conductor logs we see the following error:

2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager Traceback (most recent call last):
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/ironic/conductor/base_manager.py", line 453, in _conductor_service_record_keepalive
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager self.conductor.touch()
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/ironic/objects/conductor.py", line 116, in touch
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager self.dbapi.touch_conductor(self.hostname)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager ectxt.value = e.inner_exc
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in _exit_
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager self.force_reraise()
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager raise self.value
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager return f(*args, **kwargs)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/ironic/db/sqlalchemy/api.py", line 1209, in touch_conductor
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager 'online': True})
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3818, in update
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager update_op.exec_()
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1671, in exec_
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager self._do_exec()
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1864, in _do_exec
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager self._execute_stmt(update_stmt)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1676, in _execute_stmt
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager self.result = self.query._execute_crud(stmt, self.mapper)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3350, in _execute_crud
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager return conn.execute(stmt, self._params)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager return meth(self, multiparams, params)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager return connection._execute_clauseelement(self, multiparams, params)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager distilled_params,
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager e, statement, parameters, cursor, context
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager util.raise_from_cause(newraise, exc_info)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager reraise(type(exception), exception, tb=exc_tb, cause=cause)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager raise value.with_traceback(tb)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager cursor, statement, parameters, context
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager cursor.execute(statement, parameters)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in execute
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager result = self._query(query)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager conn.query(q)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, in query
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, in _read_query_result
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager result.read()
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069, in read
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager first_packet = self.connection._read_packet()
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager packet.raise_for_error()
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager err.raise_mysql_exception(self._data)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager raise errorclass(errno, errval)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'ironic.conductors' doesn't exist")
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager [SQL: UPDATE conductors SET updated_at=%(updated_at)s, online=%(online)s WHERE conductors.hostname = %(hostname_1)s]
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager [parameters:

{'updated_at': datetime.datetime(2022, 6, 22, 19, 25, 55, 265451), 'online': 1, 'hostname_1': '10.0.98.218'}

]
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager (Background on this error at: http://sqlalche.me/e/f405)
2022-06-22 19:25:55.268 1 ERROR ironic.conductor.base_manager