Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: None
Affects Version/s: rhel-7.9.z
Component/s: leapp-repository
Labels:

Severity:
Major

Pool Team:

sst_upgrades

Story Points:
2
Blocked:
False
Blocked Reason:

Hide

None

Show
None

Release Note Type:
If docs needed, set a value

Experience:
Architecture:

All
Bugzilla Bug:
RHBZ: 1940869

PX Impact Score:
PX Technical Impact:
PX Priority Data:
PX Review Complete:
PX Scheduling Request:
SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:
Leapp fails during the step where "yum clean all" is executed in the container through systemd-nspawn.
Usually, it happens when the repositories are not properly configured, in particular with a Satellite infrastructure.
But this is not the case here: other machines, within the same infrastructure, have been upgraded successfully.
During a test, they changed the command "yum clean all" by "yum repolist all", and then the upgrade worked. It was done directly in get_available_repo_ids() from /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/rhsm.py.

Version-Release number of selected component (if applicable):
leapp-0.12.0-1.el7_9.noarch
leapp-repository-0.13.0-2.el7_9.noarch

with the current leapp-data13.tar.gz.

How reproducible:
100% for the customer, but unable to reproduce internally.

Actual results:

2021-03-17 21:37:00.746 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Loaded plugins: enabled_repos_upload, product-id, search-disabled-repos,
2021-03-17 21:37:00.757 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:               : subscription-manager
2021-03-17 21:37:00.925 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: 
2021-03-17 21:37:00.935 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: This system is not receiving updates. You can use subscription-manager on the host to register and assign subscriptions.
2021-03-17 21:37:00.947 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: 
2021-03-17 21:37:00.960 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Uploading Enabled Repositories Report
2021-03-17 21:37:00.968 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: There are no enabled repos.
2021-03-17 21:37:00.977 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:  Run \"yum repolist all\" to see the repos you have.
2021-03-17 21:37:00.984 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:  To enable Red Hat Subscription Management repositories:
2021-03-17 21:37:00.992 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:      subscription-manager repos --enable <repo>
2021-03-17 21:37:01.0   DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:  To enable custom repositories:
2021-03-17 21:37:01.8   DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:      yum-config-manager --enable <repo>
2021-03-17 21:37:04.977 DEBUG    PID: 42336 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Command ['systemd-nspawn', '--register=no', '--quiet', '-D', '/var/lib/leapp/scratch/mounts/root_/system_overlay', '--bind=/etc/hosts:/etc/hosts', '--setenv=LEAPP_NO_RHSM=0', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el7toel8/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/el7toel8/files', '--setenv=LEAPP_UNSUPPORTED=0', '--setenv=LEAPP_EXECUTION_ID=c5498300-0716-4e78-ae4b-59bad5a9a998', '--setenv=LEAPP_HOSTNAME=XXXXXX', 'yum', 'clean', 'all'] failed with exit code 1.

Additional info:

We are suspecting a side effect of overlayfs, but currently we are unable to find some evidences.
From the strace of another machine which fails, we noticed they have at least 20 ext4 filesystems (including 15 fs which are not required during the upgrade).
During the "rollback" due to the failure, leapp is unable to remove some dirs used to mount overlays (e.g.: /var/lib/leapp/scratch/mounts/root_/system_overlay/...), there are some EBUSY (Device or resource busy).
During the "yum clean all", we can see some operations done by the subscription-manager plugin, it fills /var/cache/rhsm/profile.json and then exits without any obvious reasons. Extracting the rhsm.log, we can see a strange warning (as the file exists):
```
2021-03-17 21:26:56,281 [WARNING] subscription-manager:28483:MainThread @cert_sorter.py:194 - Installed product 69 not present in response from server.
```
=> but this doesn't seem the same issue than rhbz#1911802