-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
rhel-8.10
-
None
-
Low
-
rhel-upgrades
-
3
-
False
-
False
-
-
None
-
None
-
None
-
None
-
Unspecified
-
Unspecified
-
Unspecified
-
-
All
-
None
What were you trying to do that didn't work?
A leapp upgrade had a failure because the dnf mkcache service ran prior to the upgrade starting and was still running during the rpm_scanner portion of the upgrade.
Notice the times from /var/log/dnf.log:
2025-11-07T09:12:57-0500 DDEBUG Command: dnf makecache --timer 2025-11-07T09:12:57-0500 DDEBUG Installroot: / 2025-11-07T09:12:57-0500 DDEBUG Releasever: 8 2025-11-07T09:12:57-0500 DEBUG cachedir: /var/cache/dnf 2025-11-07T09:12:57-0500 DDEBUG Base command: makecache 2025-11-07T09:12:57-0500 DDEBUG Extra commands: ['makecache', '--timer'] 2025-11-07T09:12:57-0500 DEBUG Making cache files for all metadata files. 2025-11-07T09:12:57-0500 DEBUG rhel-8-for-x86_64-appstream-rpms: has expired and will be refreshed. 2025-11-07T09:12:57-0500 DEBUG satellite-6.16-for-rhel-8-x86_64-rpms: has expired and will be refreshed. 2025-11-07T09:12:57-0500 DEBUG rhel-8-for-x86_64-baseos-rpms: has expired and will be refreshed. 2025-11-07T09:12:57-0500 DEBUG satellite-maintenance-6.16-for-rhel-8-x86_64-rpms: has expired and will be refreshed. 2025-11-07T09:12:58-0500 DEBUG repo: downloading from remote: rhel-8-for-x86_64-appstream-rpms 2025-11-07T09:14:06-0500 DEBUG rhel-8-for-x86_64-appstream-rpms: using metadata from Thu 06 Nov 2025 07:50:19 AM EST. 2025-11-07T09:14:06-0500 DEBUG repo: downloading from remote: satellite-6.16-for-rhel-8-x86_64-rpms 2025-11-07T09:14:09-0500 DEBUG satellite-6.16-for-rhel-8-x86_64-rpms: using metadata from Wed 05 Nov 2025 10:03:43 PM EST. 2025-11-07T09:14:09-0500 DEBUG repo: downloading from remote: rhel-8-for-x86_64-baseos-rpms 2025-11-07T09:15:30-0500 DEBUG rhel-8-for-x86_64-baseos-rpms: using metadata from Fri 07 Nov 2025 08:30:45 AM EST. 2025-11-07T09:15:30-0500 DEBUG repo: downloading from remote: satellite-maintenance-6.16-for-rhel-8-x86_64-rpms 2025-11-07T09:15:31-0500 DEBUG satellite-maintenance-6.16-for-rhel-8-x86_64-rpms: using metadata from Wed 05 Nov 2025 10:04:58 PM EST. 2025-11-07T09:15:32-0500 DEBUG User-Agent: constructed: 'libdnf (Red Hat Enterprise Linux 8.10; generic; Linux.x86_64)' 2025-11-07T09:15:37-0500 DDEBUG timer: sack setup: 159381 ms 2025-11-07T09:15:37-0500 DEBUG Completion plugin: Generating completion cache... 2025-11-07T09:15:37-0500 INFO Metadata cache created. 2025-11-07T09:15:37-0500 DDEBUG Cleaning up. 2025-11-07T09:15:37-0500 DDEBUG Plugins were unloaded.
Then the leapp upgrade starts here:
$ head -1 var/log/leapp/leapp-upgrade.log 2025-11-07 09:15:00.908 INFO PID: 68768 leapp: Logging has been initialized
We can see that the leapp upgrade ran after the dnf makecache was started, but then dnf makecache ended and removed the lock file.
Here is the error timestamp:
2025-11-07 09:15:00.908 INFO PID: 68768 leapp: Logging has been initialized and the error is here: 2025-11-07 09:15:36.964 ERROR PID: 68768 leapp.workflow.FactsCollection: Actor rpm_scanner has crashed: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/leapp/repository/actor_definition.py", line 75, in _do_run actor_instance.run(*args, **kwargs) File "/usr/lib/python3.6/site-packages/leapp/actors/__init__.py", line 319, in run self.process(*args) File "/etc/leapp/repos.d/system_upgrade/common/actors/rpmscanner/actor.py", line 20, in process rpmscanner.process() File "/etc/leapp/repos.d/system_upgrade/common/actors/rpmscanner/libraries/rpmscanner.py", line 111, in process rpm_streams = map_modular_rpms_to_modules() File "/etc/leapp/repos.d/system_upgrade/common/actors/rpmscanner/libraries/rpmscanner.py", line 87, in map_modular_rpms_to_modules modules = module_lib.get_modules() File "/etc/leapp/repos.d/system_upgrade/common/libraries/module.py", line 57, in get_modules base = _create_or_get_dnf_base(base) File "/etc/leapp/repos.d/system_upgrade/common/libraries/module.py", line 47, in _create_or_get_dnf_base base.fill_sack() File "/usr/lib/python3.6/site-packages/dnf/base.py", line 429, in fill_sack self.repos.all().disable() File "/usr/lib/python3.6/site-packages/dnf/lock.py", line 147, in __exit__ os.unlink(self.target) FileNotFoundError: [Errno 2] No such file or directory: '/var/cache/dnf/metadata_lock.pid'
dnf makecache is a service that is automatically ran by a timer service, it doesn't show in the dnf history:
$ cat etc/systemd/system/timers.target.wants/dnf-makecache.timer [Unit] Description=dnf makecache --timer ConditionKernelCommandLine=!rd.live.image # See comment in dnf-makecache.service ConditionPathExists=!/run/ostree-booted Wants=network-online.target [Timer] OnBootSec=10min OnUnitInactiveSec=1h RandomizedDelaySec=60m Unit=dnf-makecache.service [Install] WantedBy=timers.target
The issue is hard to reproduce since it's a timing issue.
Also I found this upstream bug, this makecache seems to explain why the issue happens inconsistently.
https://github.com/oamg/leapp-repository/issues/250
What is the impact of this issue to you?
I think it would be useful to have some sort of a check to make sure the dnf-makecache service isn't running at the beginning of a leapp upgrade, and then allow it some time to complete (minutes before giving inhibitor).
Please provide the package NVR for which the bug is seen:
leapp-0.19.0-1.el8_10.noarch
How reproducible is this bug?:
Not easily, since it is a timing issue but caught data of the issue in the linked case. You can also see from testing dnf makecache that it does create and remove the `/var/cache/dnf/metadata_lock.pid` during its run. If it happens to remove this file as the rpm_scanner actor is running, it leads to this issue.
- links to
