Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-9900

nova-manage db purge fails on large datasets

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • rhos-18.0.1
    • rhos-18.0.0
    • openstack-nova
    • None
    • Moderate

      From https://bugzilla.redhat.com/show_bug.cgi?id=2280606:

      Description of problem:
      One of our customers faced upstream bug https://bugs.launchpad.net/nova/+bug/1983188 in his RHOSP 16.2 deployment when running the following command against nova DB with 18 GBs of records in shadow_instances table:

      1. podman exec -it nova_api_cron bash -c "nova-manage db purge --verbose --all --all-cells"
        An error has occurred:
        Traceback (most recent call last):
        File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 761, in _commit_impl
        self.engine.dialect.do_commit(self.connection)
        File "/usr/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2215, in do_commit
        dbapi_connection.commit()
        File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit
        self._read_ok_packet()
        File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
        pkt = self._read_packet()
        File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet
        packet.raise_for_error()
        File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error
        err.raise_mysql_exception(self._data)
        File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
        raise errorclass(errno, errval)
        pymysql.err.OperationalError: (1180, 'Got error 90 "Message too long" during COMMIT')

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/nova/cmd/manage.py", line 3314, in main
      ret = fn(*fn_args, **fn_kwargs)
      File "/usr/lib/python3.6/site-packages/nova/cmd/manage.py", line 747, in purge
      status_fn=status)
      File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 5817, in purge_shadow_tables
      deleted = conn.execute(delete)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
      return meth(self, multiparams, params)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
      return connection._execute_clauseelement(self, multiparams, params)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
      distilled_params,
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1272, in _execute_context
      self._root._commit_impl(autocommit=True)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 763, in _commit_impl
      self._handle_dbapi_exception(e, None, None, None, None)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception
      util.raise_from_cause(newraise, exc_info)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
      reraise(type(exception), exception, tb=exc_tb, cause=cause)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
      raise value.with_traceback(tb)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 761, in _commit_impl
      self.engine.dialect.do_commit(self.connection)
      File "/usr/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2215, in do_commit
      dbapi_connection.commit()
      File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit
      self._read_ok_packet()
      File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
      pkt = self._read_packet()
      File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet
      packet.raise_for_error()
      File "/usr/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error
      err.raise_mysql_exception(self._data)
      File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
      raise errorclass(errno, errval)
      sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got error 90 "Message too long" during COMMIT')
      (Background on this error at: http://sqlalche.me/e/e3q8)

      I understand that it is not going to be fixed in RHOSP 16.2, but kindly ask engineering to consider fixing it in RHOSP 17.1.

      Version-Release number of selected component (if applicable):
      RHOSP 16.2 +

      How reproducible: see description

      Actual results: nova-manage fails to purge archived entries if their number is too big

      Expected results: nova-manage successfully drops archived entries no matter how big their number is

            mwitt@redhat.com melanie witt
            mwitt@redhat.com melanie witt
            rhos-dfg-compute
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: