[leapp] IPU 7->8: a subset of packages are not downloaded in the DNF cache

Description of problem:
A customer was not able to in-place upgrade its Satellite server (IPU 7->8, using CDN repos).
Leapp did not report significant warnings and didn't inhibit the upgrade.
After the reboot, the system falls in the emergency shell, a subset of the required packages (a few hundreds) having not been downloaded in the DNF cache.

After many checks, Satellite team confirmed there was no issue in their configuration (hence using the more generic RHEL product for this issue).
The rest of the configuration seem also standard, nothing obvious can explain what happened.
There is likely a small thing we missed as this issue has never been seen.
From my understanding, the result seems a misbehavior of the underlying DNF plugin used by Leapp (for example such an error message can be seen with by running "dnf reinstall --downloadonly" with a package name which is not installed).

From leapp-upgrade.log:
2023-01-11 15:20:41.991 DEBUG PID: 17444 leapp.workflow.Download.dnf_package_download: Package kbd-misc available, but not installed.
2023-01-11 15:20:41.998 DEBUG PID: 17444 leapp.workflow.Download.dnf_package_download: Package katello-client-bootstrap available, but not installed.
2023-01-11 15:20:42.27 DEBUG PID: 17444 leapp.workflow.Download.dnf_package_download: Package iputils available, but not installed.
...

$ grep -c dnf_package_download:.Package.*but.not.installed 0210-03391250_leap.tgz/var/log/leapp/leapp-upgrade.log
628

Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.17.0-1.el7_9

How reproducible:
Not able to reproduce internally.

Actual results:
(From rdsosreport.txt)

[ 271.382608] localhost upgrade[1337]: Transaction Summary
[ 271.382608] localhost upgrade[1337]: ======================================================================================================================================================
==============
[ 271.382608] localhost upgrade[1337]: Install 778 Packages
[ 271.382608] localhost upgrade[1337]: Upgrade 594 Packages
[ 271.382608] localhost upgrade[1337]: Remove 538 Packages
[ 271.382608] localhost upgrade[1337]: Downgrade 35 Packages
[ 271.382608] localhost upgrade[1337]: Total size: 1.2 G
[ 271.382608] localhost upgrade[1337]: Total download size: 248 M
[ 271.382608] localhost upgrade[1337]: Downloading Packages:
[ 271.382608] localhost upgrade[1337]: STDERR:
[ 271.382608] localhost upgrade[1337]: The following modules were requested to be enabled, but they are unavailable: virt:av
[ 271.382608] localhost upgrade[1337]: Warning: Package marked by Leapp to install not found in repositories metadata: rubygem-foreman_ansible_core rubygem-foreman-tasks-core boost-python3-
devel boost-python3 rubygem-foreman_remote_execution_core log4j12
[ 271.382608] localhost upgrade[1337]: Warning: Package marked by Leapp to upgrade not found in repositories metadata: python2-leapp gpg-pubkey leapp leapp-upgrade-el7toel8
[ 271.382608] localhost upgrade[1337]: Error opening /var/cache/dnf/rhel-8-for-x86_64-baseos-rpms-f01df491f8fd5ba3/packages/efibootmgr-16-1.el8.x86_64.rpm: No such file or directory
[ 271.382608] localhost upgrade[1337]: Package "efibootmgr-16-1.el8.x86_64" from repository "rhel-8-for-x86_64-baseos-rpms" has incorrect checksum
[ 271.393330] localhost upgrade[1337]: Error opening /var/cache/dnf/rhel-8-for-x86_64-baseos-rpms-f01df491f8fd5ba3/packages/libcgroup-0.41-19.el8.x86_64.rpm: No such file or directory
[ 271.393330] localhost upgrade[1337]: Package "libcgroup-0.41-19.el8.x86_64" from repository "rhel-8-for-x86_64-baseos-rpms" has incorrect checksum
[ 271.393330] localhost upgrade[1337]: Error opening /var/cache/dnf/rhel-8-for-x86_64-baseos-rpms-f01df491f8fd5ba3/packages/python3-iniparse-0.4-31.el8.noarch.rpm: No such file or directory
[ 271.393330] localhost upgrade[1337]: Package "python3-iniparse-0.4-31.el8.noarch" from repository "rhel-8-for-x86_64-baseos-rpms" has incorrect checksum
:
[ 272.623993] localhost upgrade[1337]: Error: Some packages have invalid cache, but cannot be downloaded due to "--cacheonly" option

Additional info:

• Making a diff between the list of RPMs already downloaded into the upgrade container's DNF cache (/var/lib/leapp/el8userspace/var/dnf/cache) and the packages listed below gives the missing RPMs that have to be downloaded again.

$ awk '/will.be.(a.downgrade|installed|an.upgrade)/

' 0290-rdsosreport.txt_1601_1.txt | wc -l
2810

• The workaround was to chroot the upgrade container, enable a few modules and then use "dnf install --downloadonly" to fill up the DNF cache with the missing packages (list of pkgs below is specific to the customer installation).

1. mount -t proc none /var/lib/leapp/el8userspace/proc
2. mount -t sysfs none /var/lib/leapp/el8userspace/sys
3. mount -o bind /dev /var/lib/leapp/el8userspace/dev
4. chroot /var/lib/leapp/el8userspace

1. dnf --nogpgcheck --setopt=keepcache=1 --releasever=8.6 --setopt=module_platform_id=platform:el8 --disablerepo=* --enablerepo=rhel-8-for-x86_64-baseos-rpms --enablerepo=rhel-8-for-x86_64-appstream-rpms --enablerepo=rhel-8-for-x86_64-highavailability-rpms --enablerepo=satellite-6.11-for-rhel-8-x86_64-rpms --enablerepo=satellite-maintenance-6.11-for-rhel-8-x86_64-rpms --enablerepo=advanced-virt-for-rhel-8-x86_64-rpms module enable postgresql:12 ruby:2.7 satellite

1. dnf --nogpgcheck --setopt=keepcache=1 --releasever=8.6 --setopt=module_platform_id=platform:el8 --disablerepo=* --enablerepo=rhel-8-for-x86_64-baseos-rpms --enablerepo=rhel-8-for-x86_64-appstream-rpms --enablerepo=rhel-8-for-x86_64-highavailability-rpms --enablerepo=satellite-6.11-for-rhel-8-x86_64-rpms --enablerepo=satellite-maintenance-6.11-for-rhel-8-x86_64-rpms --enablerepo=advanced-virt-for-rhel-8-x86_64-rpms --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms install --downloadonly alsa-lib ansible-collection-redhat-satellite ansible-collection-redhat-satellite_operations ansiblerole-foreman_scap_client ansiblerole-insights-client ansible-runner ansible-test apache-commons-codec apache-commons-collections apache-commons-lang apache-commons-logging apr apr-util atk audit bc bind-libs bind-libs-lite bind-license bind-utils boost boost-atomic boost-chrono boost-context boost-date-time boost-devel boost-filesystem boost-graph boost-iostreams boost-locale boost-math boost-program-options boost-random boost-regex boost-serialization boost-signals boost-system boost-test boost-thread boost-timer boost-wave bzip2 cairo cairo-gobject candlepin candlepin-selinux c-ares checkpolicy chrony copy-jdk-configs createrepo_c createrepo_c-libs cronie cronie-anacron crontabs cyrus-sasl cyrus-sasl-gssapi cyrus-sasl-plain daxctl-libs dejavu-fonts-common dejavu-sans-fonts desktop-file-utils dmidecode dosfstools dwz dynflow-utils e2fsprogs efibootmgr efivar-libs elfutils emacs-filesystem ethtool fio fipscheck fipscheck-lib firewalld firewalld-filesystem fontconfig fontpackages-filesystem foreman foreman-bootloaders-redhat foreman-bootloaders-redhat-tftpboot foreman-cli foreman-debug foreman-dynflow-sidekiq foreman-ec2 foreman-gce foreman-libvirt foreman-openstack foreman-ovirt foreman-postgresql foreman-proxy foreman-selinux foreman-service foreman-vmware freetype fribidi gdb gdk-pixbuf2 genisoimage geoipupdate giflib git glib2-devel glib-networking gobject-introspection gpm-libs graphite2 groff-base grub2-efi-ia32-modules grub2-efi-x64 grub2-efi-x64-modules grub2-pc grub2-pc-modules grub2-tools-extra gsettings-desktop-schemas gtk2 gtk-update-icon-cache harfbuzz hicolor-icon-theme httpd httpd-tools http-parser initscripts insights-client ipmitool ipset ipset-libs iptables irqbalance jasper-libs java-11-openjdk java-11-openjdk-headless java-1.8.0-openjdk java-1.8.0-openjdk-headless javapackages-tools jbigkit-libs jss katello-certs-tools katello-client-bootstrap katello-common katello-debug katello-selinux kernel-tools kernel-tools-libs leapp-deps less libcgroup libdb-cxx libdhash libdrm libecap libestr libfastjson libffi-devel libfontenc libglvnd libglvnd-egl libglvnd-glx libibverbs libICE libidn libipa_hbac libjpeg-turbo libkadm5 libmodman libnetfilter_conntrack libnfnetlink libpcap libpipeline libpmem libpmemblk libpng libproxy librados2 librbd1 librdmacm libSM libsmbclient libsodium libsoup libss libsss_autofs libsss_certmap libsss_idmap libsss_nss_idmap libsss_sudo libsysfs libthai libtiff libtool-ltdl libusal libuser libuuid-devel libvirt-libs libwayland-client libwayland-server libwebp libwebsockets libX11 libX11-common libXau libxcb libXcomposite libXcursor libXdamage libXext libXfixes libXft libXi libXinerama libXrandr libXrender libxshmfence libxslt libXtst libXxf86vm lksctp-tools lshw lsof lua lz4 lzo mailcap make man-db mesa-libEGL mesa-libgbm mesa-libGL mesa-libglapi microcode_ctl mod_ssl mokutil mtools ndctl-libs net-snmp-libs net-tools newt nmap-ncat nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-tools nss-util numactl-libs OpenIPMI OpenIPMI-libs openscap openscap-scanner openssh-server ostree pango parted passwd patch pciutils pcre-devel pcsc-lite-libs perl perl-Carp perl-Compress-Raw-Bzip2 perl-Compress-Raw-Zlib perl-constant perl-Data-Dumper perl-DBI perl-Digest perl-Digest-MD5 perl-Encode perl-Error perl-Exporter perl-File-Path perl-File-Temp perl-Filter perl-Getopt-Long perl-Git perl-HTTP-Tiny perl-IO-Compress perl-libs perl-macros perl-parent perl-PathTools perl-Pod-Escapes perl-podlators perl-Pod-Perldoc perl-Pod-Simple perl-Pod-Usage perl-Scalar-List-Utils perl-Socket perl-srpm-macros perl-Storable perl-TermReadKey perl-Text-ParseWords perl-Thread-Queue perl-threads perl-threads-shared perl-Time-HiRes perl-Time-Local pixman polkit polkit-pkla-compat postfix psacct pulpcore-selinux puppet-agent puppet-agent-oauth puppet-foreman_scap_client puppetlabs-stdlib puppetserver pygobject2 python2-jinja2 python2-markupsafe python2-rpm-macros python3-attrs python3-cairo python3-cffi python3-chardet python3-click python3-cryptography python3-gobject python3-gobject-base python3-idna python3-iniparse python3-jinja2 python3-jsonschema python3-lxml python3-markupsafe python3-pip python3-productmd python3-psycopg2 python3-pycparser python3-pymongo python3-pyOpenSSL python3-pyparsing python3-pytz python3-requests python3-setuptools python3-subscription-manager-rhsm python3-uritemplate python3-urllib3 python3-webencodings python-rpm-macros python-srpm-macros qpid-cpp-client qpid-cpp-client-devel qpid-cpp-server qpid-cpp-server-linearstore qpid-dispatch-router qpid-proton-c qpid-qmf qpid-tools qrencode-libs rdma-core redhat-access-insights-puppet redhat-logos redhat-rpm-config redhat-support-lib-python redhat-support-tool rhn-check rhn-client-tools rhnlib rhnsd rhn-setup rootfiles rpm-build rsync rsyslog ruby rubygem-bigdecimal rubygem-clamp rubygem-foreman_scap_client rubygem-highline rubygem-io-console rubygem-json rubygem-psych rubygem-rdoc rubygems ruby-libs samba-libs saslwrapper satellite satellite-cli satellite-common satellite-maintain scap-security-guide scl-utils selinux-policy selinux-policy-targeted sg3_utils sg3_utils-libs shim-ia32 shim-x64 slang snappy sos sqlite sshpass sssd sssd-ad sssd-client sssd-common sssd-common-pac sssd-ipa sssd-krb5 sssd-krb5-common sssd-ldap sssd-proxy strace subscription-manager subscription-manager-rhsm-certificates sudo syslinux syslinux-tftpboot tftp-server tmux tomcatjss tree ttmkfdir tuned tzdata-java unzip usermode uuid vim-common vim-enhanced vim-filesystem vim-minimal virt-what wget xalan-j2 xdg-utils xerces-j2 xfsprogs xinetd xml-common xml-commons-apis xml-commons-resolver xorg-x11-fonts-Type1 xorg-x11-font-utils yajl yum yum-utils zip

1. exit
2. umount /var/lib/leapp/el8userspace/dev
3. umount /var/lib/leapp/el8userspace/sys
4. umount /var/lib/leapp/el8userspace/proc