Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-10783

Adoption's mysqlcheck sometimes fails

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ?
    • ?
    • None
    • Hide
      .Possible database error before adoption

      You might see a database error for system table `mysql.proc` if you run `mysqlcheck` before adopting the OSP database.

      ----
      [...]
      mysql.plugin OK
      mysql.proc Needs upgrade
      mysql.procs_priv OK
      [...]
      ----

      This error message is harmless and results from a system table's redo log that was not replicated correctly when the galera cluster was bootstrapped.

      *Workaround:* You can remove the error by repairing the `mysql.proc` system table:

      .Example command
      ----
      oc run mariadb-client ${MARIADB_CLIENT_ANNOTATIONS} -q --image ${MARIADB_IMAGE} -i --rm --restart=Never -- \
          mysql -h $SOURCE_MARIADB_IP -u root -p"$SOURCE_DB_ROOT_PASSWORD" -e "repair table mysql.proc;"
      ----

      .Example output
      ----
      +------------+--------+----------+---------------------------------+
      | Table | Op | Msg_type | Msg_text |
      +------------+--------+----------+---------------------------------+
      | mysql.proc | repair | info | Running zerofill on moved table |
      | mysql.proc | repair | status | OK |
      +------------+--------+----------+---------------------------------+
      ----

      The table and its redo log are fixed and replicated across all galera nodes. Re-run the mysqlcheck and continue the adoption procedure.
      Show
      .Possible database error before adoption You might see a database error for system table `mysql.proc` if you run `mysqlcheck` before adopting the OSP database. ---- [...] mysql.plugin OK mysql.proc Needs upgrade mysql.procs_priv OK [...] ---- This error message is harmless and results from a system table's redo log that was not replicated correctly when the galera cluster was bootstrapped. *Workaround:* You can remove the error by repairing the `mysql.proc` system table: .Example command ---- oc run mariadb-client ${MARIADB_CLIENT_ANNOTATIONS} -q --image ${MARIADB_IMAGE} -i --rm --restart=Never -- \     mysql -h $SOURCE_MARIADB_IP -u root -p"$SOURCE_DB_ROOT_PASSWORD" -e "repair table mysql.proc;" ---- .Example output ---- +------------+--------+----------+---------------------------------+ | Table | Op | Msg_type | Msg_text | +------------+--------+----------+---------------------------------+ | mysql.proc | repair | info | Running zerofill on moved table | | mysql.proc | repair | status | OK | +------------+--------+----------+---------------------------------+ ---- The table and its redo log are fixed and replicated across all galera nodes. Re-run the mysqlcheck and continue the adoption procedure.
    • Known Issue
    • Done
    • Moderate

      We've observed that the mysqlcheck on source DB during adoption can sometimes fail. Initially we suspected incompatibility between the running DB version and the version of mariadb in the container image used for the check, but this is likely not the cause, because the the failure is intermittent.

      See the discussion at:

      https://github.com/openstack-k8s-operators/data-plane-adoption/pull/654

      And at:

      https://redhat-internal.slack.com/archives/C03MD4LG22Z/p1729238412855609

      Also noting that this issue was originally discussed and addressed as a CIX with https://issues.redhat.com/browse/OSPCIX-521

      Attempted workaround by re-running mysqlcheck multiple times did not seem to work:
       

      TASK [get_services_configuration : run mysqlcheck on the original DB to look for things that are not OK] ***
FAILED - RETRYING: [localhost]: run mysqlcheck on the original DB to look for things that are not OK (6 retries left).
FAILED - RETRYING: [localhost]: run mysqlcheck on the original DB to look for things that are not OK (5 retries left).
FAILED - RETRYING: [localhost]: run mysqlcheck on the original DB to look for things that are not OK (4 retries left).
FAILED - RETRYING: [localhost]: run mysqlcheck on the original DB to look for things that are not OK (3 retries left).
FAILED - RETRYING: [localhost]: run mysqlcheck on the original DB to look for things that are not OK (2 retries left).
FAILED - RETRYING: [localhost]: run mysqlcheck on the original DB to look for things that are not OK (1 retries left).

      Running `mysql_upgrade` currently seems to work but this is not something that can be recommended in the docs because we're dealing with the source DB environment which we should treat as read-only.

              rhn-engineering-dciabrin Damien Ciabrini
              jstransk@redhat.com Jiri Stransky
              rhos-dfg-pidone
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: