Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-55871

leapp in-place upgrades 8 -> 9 crashes with "sqlite3.OperationalError: unable to open database file"

XMLWordPrintable

    • leapp-0.18.0-1.el8_10
    • No
    • None
    • 1
    • rhel-sst-upgrades
    • 26
    • 1
    • False
    • Hide

      None

      Show
      None
    • None
    • Leapp: 9.5 & 10.0 public beta
    • None

      The in-place upgrade (IPU) from RHEL 8 to RHEL 9 fails with following errors:

      2024-08-22 14:29:11.45  DEBUG    PID: 37068 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['systemd-nspawn', '--register=no', '--quiet', '--keep-unit', '--capability=all', '-D', '/var/lib/leapp/el9userspace', '--setenv=LEAPP_DEVEL_USE_PERSISTENT_PACKAGE_CACHE=1', '--setenv=LEAPP_NO_RHSM=1', '--setenv=LEAPP_UNSUPPORTED=1', '--setenv=LEAPP_HOSTNAME=leapp-20240822144959', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_NO_RHSM_FACTS=1', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.4', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_IPU_IN_PROGRESS=8to9', '--setenv=LEAPP_EXECUTION_ID=ecc3e7fd-ef11-49bb-9c74-5fee4942afc9', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el8toel9/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files', '--setenv=LEAPP_DEVEL_DATABASE_SYNC_OFF=1', 'rpm', '-qf', '/etc/pki/ca-trust/extracted/pem/directory-hash/ca-bundle.crt']


warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
Process Process-448:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 185, in run
    stdin=stdin, env=env, encoding=encoding)
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 199, in _call
    **extra
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 73, in _multiplex
    callback_linebuffered((fd, fd_type), pre)
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 143, in _logfile_logging_handler
    api.current_logger().debug(line)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1296, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
    self.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle
    self.callHandlers(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle
    self.emit(record)
  File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 40, in emit
    self._do_emit(log_data)
  File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 45, in _do_emit
    Audit(**log_data).store()
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 87, in store
    with get_connection(db) as connection:
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 73, in get_connection
    return create_connection(cfg.get('database', 'path'))
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 26, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 60, in create_connection
    return _initialize_database(sqlite3.connect(path))
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 27, in _initialize_database
    schema_version = db.execute('PRAGMA schema_version').fetchone()[0]
sqlite3.OperationalError: unable to open database file

      Note that in this particular ticket we track the traceback which is raised right after many logs produced by single actor saying: External command has started. You can see example of such a log in the top of the snippet above.

      You can hit similar tracebacks also in other cases, but they can be connected to a different issue RHEL-26459 which is generic problem affected by the system setup. Both issues have same workaround however (below).

      Root cause

      The leapp framework v0.17.0 and lower has bug inside the standard library. When calling the `run` method to execute a shell command, each call leaves opened 2 file descriptors until the end of the actor execution. In case that one actor has enough number of such calls, it will reach the limit of enabled opened FDs on the system which breaks the leapp execution completely.

      Workaround

      Increase the limit of opened file descriptors, like this:

      ulimit -n 16384

      Additional info

      • Note that for RHEL 8.10.z this issue is going to be covered by RHEL-27848 - rebase of leapp framework to version 0.18.0

              leapp-notifications leapp-notifications
              pstodulk@redhat.com Petr Stodulka
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: