Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: rhel-8.10.z
Affects Version/s: rhel-8.8.0, rhel-8.10
Component/s: leapp
Labels:

Fixed in Build:
leapp-0.18.0-1.el8_10
Regression:
No
Severity:
None
sprint_count:
1

Pool Team:

sst_upgrades

Internal Target Milestone:
26
Story Points:
1
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
Leapp: 9.5 & 10.0 public beta

Git Pull Request:
https://github.com/oamg/leapp/pull/880
Preliminary Testing:
Requested
Errata Link:
https://errata.engineering.redhat.com/advisory/132709
Test Coverage:
None

Experience:

PX Impact Score:
PX Technical Impact:
PX Impact Range:
PX Priority Data:
PX Review Complete:
SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

The in-place upgrade (IPU) from RHEL 8 to RHEL 9 fails with following errors:

2024-08-22 14:29:11.45  DEBUG    PID: 37068 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['systemd-nspawn', '--register=no', '--quiet', '--keep-unit', '--capability=all', '-D', '/var/lib/leapp/el9userspace', '--setenv=LEAPP_DEVEL_USE_PERSISTENT_PACKAGE_CACHE=1', '--setenv=LEAPP_NO_RHSM=1', '--setenv=LEAPP_UNSUPPORTED=1', '--setenv=LEAPP_HOSTNAME=leapp-20240822144959', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_NO_RHSM_FACTS=1', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.4', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_IPU_IN_PROGRESS=8to9', '--setenv=LEAPP_EXECUTION_ID=ecc3e7fd-ef11-49bb-9c74-5fee4942afc9', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el8toel9/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files', '--setenv=LEAPP_DEVEL_DATABASE_SYNC_OFF=1', 'rpm', '-qf', '/etc/pki/ca-trust/extracted/pem/directory-hash/ca-bundle.crt']


warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
Process Process-448:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 185, in run
    stdin=stdin, env=env, encoding=encoding)
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 199, in _call
    **extra
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 73, in _multiplex
    callback_linebuffered((fd, fd_type), pre)
  File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 143, in _logfile_logging_handler
    api.current_logger().debug(line)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1296, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
    self.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle
    self.callHandlers(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle
    self.emit(record)
  File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 40, in emit
    self._do_emit(log_data)
  File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 45, in _do_emit
    Audit(**log_data).store()
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 87, in store
    with get_connection(db) as connection:
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 73, in get_connection
    return create_connection(cfg.get('database', 'path'))
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 26, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 60, in create_connection
    return _initialize_database(sqlite3.connect(path))
  File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 27, in _initialize_database
    schema_version = db.execute('PRAGMA schema_version').fetchone()[0]
sqlite3.OperationalError: unable to open database file

Note that in this particular ticket we track the traceback which is raised right after many logs produced by single actor saying: External command has started. You can see example of such a log in the top of the snippet above.

You can hit similar tracebacks also in other cases, but they can be connected to a different issue RHEL-26459 which is generic problem affected by the system setup. Both issues have same workaround however (below).

Root cause

The leapp framework v0.17.0 and lower has bug inside the standard library. When calling the `run` method to execute a shell command, each call leaves opened 2 file descriptors until the end of the actor execution. In case that one actor has enough number of such calls, it will reach the limit of enabled opened FDs on the system which breaks the leapp execution completely.

Workaround

Increase the limit of opened file descriptors, like this:

ulimit -n 16384

Additional info

Note that for RHEL 8.10.z this issue is going to be covered by RHEL-27848 - rebase of leapp framework to version 0.18.0

is duplicated by

SAT-27977 Leapp upgrade process on a freshly installed Red Hat Satellite 6.16 on RHEL-8 fails during the preupgrade stage

Closed

is incorporated by

RHEL-27848 Rebase leapp to v0.18.0 (rhel-8.10)

Release Pending

relates to

RHEL-26459 Leapp hits the default per-process file limit after the last update

Planning

links to

RHBA-2024:132709 leapp bug fix update

mentioned in: Page Loading...; Page Loading...

(1 mentioned in)

Details

Description

Root cause

Workaround

Additional info

Attachments

Issue Links

Activity

People

Dates