Loading...

Type: Bug
Resolution: Cannot Reproduce
Priority: Undefined
Fix Version/s: None
Affects Version/s: rhel-7.9.z
Component/s: leapp-repository
Labels:
- dev-effort:enormous
- leapp-core

Regression:
None
Severity:
Critical

Pool Team:

rhel-sst-upgrades

Story Points:
None
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
None

Preliminary Testing:
None
Test Coverage:
None

Experience:
Architecture:

All

PX Impact Score:
PX Priority Data:
SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

What were you trying to do that didn't work?

In some RARE circumstances that we cannot reproduce, Leapp fails to download the required packages in the DNF cache of the el8 target userspace container.

In the case attached, it happens behind a Satellite server but we cannot confirm whether it already happened with CDN or local repos.

Before attaching any case to this Jira, double-check with the Sysmgmt team whether the repositories / Content View are correctly configured and test them with DNF as explained in

Leapp In-Place Upgrade: how to check DNF is working with the Satellite repositories from the client side? https://access.redhat.com/solutions/6978392

The DNF dry run completes without errors despite showing most of base packages are not present:

2024-04-18 12:27:22.264 DEBUG    PID: 104381 leapp.workflow.InterimPreparation.dnf_dry_run: Warning: Package marked by Leapp to upgrade not found in repositories metadata: kbd-misc iputils rhn-check perl-parent numactl-libs rsyslog gdbm gpgme rpm-devel trousers katello-host-tools bzip2-libs unzip libdb-utils iwl2030-firmware libcom_err alsa-firmware libseccomp gnutls perl-File-Temp NetworkManager-config-server perl-Text-ParseWords p11-kit libyaml device-mapper-persistent-data cryptsetup-libs libselinux-utils perl-Scalar-List-Utils parted bind-libs leapp-upgrade-el7toel8-deps pygobject2 libxslt plymouth libunistring keyutils-libs perl-Encode libgpg-error lvm2-libs perl-Data-Dumper gawk xmlsec1-openssl iperf3 jansson bind-utils libverto-devel e2fsprogs-libs mlocate iprutils file-libs grub2-tools libxcb dnf zip libnetfilter_conntrack perl tzdata net-snmp-devel iwl3160-firmware libreport-filesystem libICE libcom_err-devel perl-ExtUtils-Install setup kernel-tools-libs libtasn1 iptables sqlite tar sudo rhnsd nmap-ncat logrotate emacs-filesystem libgomp glibc-devel alsa-lib libdnet perl-constant yum-utils ncurses-libs polkit glibc-headers perl-Pod-Usage zlib-devel katello-ca-consumer-cap01alt.pjm.com postfix pciutils perl-Git gpg-pubkey openssh-clients libuser perl-libs libX11 lz4 kbd kexec-tools perl-Time-HiRes geoipupdate systemd selinux-policy subscription-manager-rhsm-certificates libacl dracut-config-rescue basesystem libnl3 libSM policycoreutils popt-devel redhat-logos iwl3945-firmware libidn grubby lzo libss krb5-devel redhat-support-tool nspr dbus sos psmisc net-snmp-libs perl-threads libnl3-cli redhat-support-lib-python vim-filesystem audit-libs nss-sysinit libgcc gettext-libs dbus-libs perl-ExtUtils-Manifest usermode bash glibc-common rootfiles libedit openssh libuuid openssl rhnlib iwl5000-firmware yum systemtap-sdt-devel bind-export-libs iwl100-firmware lvm2 perl-Time-Local xz iwl6000g2b-firmware lshw iwl7260-firmware vim-minimal checkpolicy NetworkManager-libnm microcode_ctl NetworkManager fxload iwl6000-firmware groff-base perl-Getopt-Long dos2unix libdb-devel rhn-client-tools dhcp-libs ncurses-base grep bind-libs-lite lm_sensors-devel iwl1000-firmware python-gofer cronie-anacron xfsprogs firewalld-filesystem man-db libcroco perl-Pod-Simple selinux-policy-targeted xz-libs dbus-glib crontabs net-snmp dracut-network which libcap plymouth-scripts libdb passwd kmod-libs NetworkManager-team perl-Pod-Escapes lm_sensors-libs desktop-file-utils libsmartcols rpm-libs strace sg3_utils nettle elfutils-default-yama-scope nss-tools libpipeline xmlsec1 iproute openssl-devel popt perl-Pod-Perldoc gobject-introspection libpwquality perl-macros iwl2000-firmware krb5-workstation libsepol-devel os-prober ksh ethtool kernel-headers grub2-pc systemd-libs perl-devel cpio util-linux libdaemon gdbm-devel net-snmp-utils libfastjson tuned krb5-libs net-tools teamd perl-HTTP-Tiny pcre-devel json-c libpciaccess sed perl-threads-shared libaio kmod nss-util hwdata libestr device-mapper lua perl-Carp kpartx libpng irqbalance perl-Exporter xdg-utils hardlink kernel-tools newt libsolv sysstat perl-File-Path virt-what nss-softokn lsof binutils libattr dnf-data iotop biosdevname rpm gnupg2 libgcrypt libverto net-snmp-agent-libs dracut cracklib procps-ng plymouth-core-libs ncurses libdnf telnet elfutils-devel perl-PathTools ipset-libs zlib libselinux libassuan kernel libxml2 expat leapp-upgrade-el7toel8 rhn-setup chkconfig perl-ExtUtils-MakeMaker libcurl perl-Error make hostname katello-agent coreutils libcap-ng iwl6050-firmware qpid-proton-c pam subscription-manager librhsm p11-kit-trust iwl4965-firmware pinentry perl-podlators perl-ExtUtils-ParseXS openldap libsysfs alsa-tools-firmware leapp perl-Filter librepo audit ca-certificates libsepol filesystem gzip autogen-libopts vim-common less rpm-build-libs openssl-libs python2-leapp xz-devel kbd-legacy libpcap iwl135-firmware libselinux-devel snappy gmp iwl6000g2a-firmware bind-license ipset perl-Socket libdrm polkit-pkla-compat iwl105-firmware pcre2 gofer cyrus-sasl-lib gpm-libs fuse-libs perl-TermReadKey xorg-x11-xauth shared-mime-info libblkid json-glib libmspack libmodulemd cracklib-dicts libicu rsync glibc shadow-utils libcgroup linux-firmware libXau fipscheck bc dmidecode libX11-common wget device-mapper-libs libteam vim-enhanced elfutils-libs dhcp-common acl libmount libkadm5 glib2 wpa_supplicant libstdc++ libtool-ltdl libmnl sg3_utils-libs nss iwl5150-firmware pcre perl-Test-Harness git cronie grub2-pc-modules tcpdump grub2-common perl-Storable pciutils-libs grub2-tools-extra traceroute elfutils-libelf-devel openssh-server libXt libXext python-qpid-proton info libnfnetlink leapp-deps mtr device-mapper-event-libs firewalld initscripts python-gofer-proton device-mapper-event libsemanage libndp libcomps mailx e2fsprogs file curl findutils lsscsi bzip2 freetype libXmu gettext elfutils-libelf nss-softokn-freebl fipscheck-lib open-vm-tools libutempter qrencode-libs katello-host-tools-fact-plugin libffi readline slang ustr NetworkManager-tui diffutils fuse grub2-tools-minimal keyutils-libs-devel

After the reboot on the RHEL-Upgrade-Initramfs grub entry, the packages listed above are not found while trying to proceed with the real upgrade transaction. Example:

[   49.875772] localhost upgrade[955]:             Package "NetworkManager-team-1:1.40.16-15.el8_9.x86_64" from repository "rhel-8-for-x86_64-baseos-rpms" has incorrect checksum
[   49.875772] localhost upgrade[955]:             Error opening /var/cache/dnf/rhel-8-for-x86_64-baseos-rpms-4911d44fccec33de/packages/NetworkManager-tui-1.40.16-15.el8_9.x86_64.rpm: No such file or directory

Checking within the el8 container, it appears dnf repolist works:

# systemd-nspawn --register=no -D /var/lib/leapp/el8userspace /usr/bin/dnf repolist --disablerepo '*' --enablerepo rhel-8-for-x86_64-baseos-rpms --enablerepo rhel-8-for-x86_64-appstream-rpms -v --releasever=8.9
Spawning container el8userspace on /var/lib/leapp/el8userspace.
Press ^] three times within 1s to kill container.
Failed to reset audit login UID. This probably means that your kernel is too
old and you have audit enabled. Note that the auditing subsystem is known to
be incompatible with containers on old kernels. Please make sure to upgrade
your kernel or to off auditing with 'audit=0' on the kernel command line before
using systemd-nspawn. Sleeping for 5s... (Operation not permitted)
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, rhel-upgrade, system-upgrade
DNF version: 4.7.0
cachedir: /var/cache/dnf
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                 86 MB/s |  67 MB     00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                              88 MB/s |  60 MB     00:00
Last metadata expiration check: 0:00:10 ago on Thu Apr 18 09:00:31 2024.
Repo-id            : rhel-8-for-x86_64-appstream-rpms
Repo-name          : Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
Repo-revision      : 1712260085
Repo-updated       : Thu Apr  4 15:48:05 2024
Repo-pkgs          : 36029
Repo-available-pkgs: 31310
Repo-size          : 115 G
Repo-baseurl       : https://capsule.customercorp.com/pulp/repos/CustomerCorp/Development/CustomerCorp_728_Leapp_Compo_CV/content/dist/rhel8/8.9/x86_64/appstream/os
Repo-expire        : 1 second(s) (last: Thu Apr 18 09:00:31 2024)
Repo-filename      : /etc/yum.repos.d/redhat.repoRepo-id            : rhel-8-for-x86_64-baseos-rpms
Repo-name          : Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
Repo-revision      : 1712081556
Repo-updated       : Tue Apr  2 14:12:36 2024
Repo-pkgs          : 15941
Repo-available-pkgs: 15935
Repo-size          : 30 G
Repo-baseurl       : https://capsule.CustomerCorp.com/pulp/repos/CustomerCorp/Development/CustomerCorp_728_Leapp_Compo_CV/content/dist/rhel8/8.9/x86_64/baseos/os
Repo-expire        : 1 second(s) (last: Thu Apr 18 09:00:21 2024)
Repo-filename      : /etc/yum.repos.d/redhat.repo
Total packages: 51970
Container el8userspace exited successfully.

~~As a workaround, packages have been "manually" downloaded using DNF within the container in question and it worked after the reboot.~~

Important: in the specific case attached here, we were obliged to remove a gpg-pubkey rpm due to the below error:

system_upgrade/common/tools/removerpmgpgkeys gpg-pubkey-2fa658e0-45700c69
failed with exit code 1.

We cannot tell for now whether it's related or not.

Please provide the package NVR for which bug is seen:

leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch

How reproducible:

Repeatable for the customer. As per their claims, it worked on many other servers within the same infrastructure.

Details

Description

What were you trying to do that didn't work?

Please provide the package NVR for which bug is seen:

How reproducible:

Attachments

Easy Agile Planning Poker

Activity

People

Dates