+++ This bug was initially created as a clone of Bug #2143451 +++

Description of problem:
satellite-maintain returns "Nothing to update, can't find new version of satellite-maintain." when attempting dry run of upgrade from 6.11 to 6.12.

Version-Release number of selected component (if applicable):
6.12

How reproducible:
Always

Steps to Reproduce:
1. Enable satellite maintenance 6.12 repo:
subscription-manager repos --enable satellite-maintenance-6.12-for-rhel-8-x86_64-rpms
2. Run satellite-maintain:
satellite-maintain upgrade check --target-version 6.12

Actual results:
[root@ip-172-31-29-4 ~]# subscription-manager repos --list-enabled
----------------------------------------------------------
Available Repositories in /etc/yum.repos.d/redhat.repo
----------------------------------------------------------
Repo ID: satellite-6.11-for-rhel-8-x86_64-rpms
Repo Name: Red Hat Satellite 6.11 for RHEL 8 x86_64 (RPMs)
Repo URL: https://cdn.redhat.com/content/dist/layered/rhel8/x86_64/satellite/6.11/os
Enabled: 1

Repo ID: rhel-8-for-x86_64-appstream-rpms
Repo Name: Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
Repo URL: https://cdn.redhat.com/content/dist/rhel8/$releasever/x86_64/appstream/os
Enabled: 1

Repo ID: rhel-8-for-x86_64-baseos-rpms
Repo Name: Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
Repo URL: https://cdn.redhat.com/content/dist/rhel8/$releasever/x86_64/baseos/os
Enabled: 1

Repo ID: satellite-maintenance-6.12-for-rhel-8-x86_64-rpms
Repo Name: Red Hat Satellite Maintenance 6.12 for RHEL 8 x86_64 (RPMS)
Repo URL: https://cdn.redhat.com/content/dist/layered/rhel8/x86_64/sat-maintenance/6.12/os
Enabled: 1

Repo ID: satellite-maintenance-6.11-for-rhel-8-x86_64-rpms
Repo Name: Red Hat Satellite Maintenance 6.11 for RHEL 8 x86_64 (RPMs)
Repo URL: https://cdn.redhat.com/content/dist/layered/rhel8/x86_64/sat-maintenance/6.11/os
Enabled: 1

[root@ip-172-31-29-4 ~]# satellite-maintain upgrade check --target-version 6.12
Checking for new version of satellite-maintain...
Security: kernel-core-4.18.0-425.3.1.el8.x86_64 is an installed security update
Security: kernel-core-4.18.0-372.32.1.el8_6.x86_64 is the currently running version
Nothing to update, can't find new version of satellite-maintain.
Can't upgrade to 6.12
Possible target versions are:
6.11.z

Expected results:
6.12 is available.

— Additional comment from on 2022-11-17T01:14:51Z

The workaround is to enable the satellite 6.12 repository (or Capsule repository if it's a Capsule) after enabling the maintenance repository:

subscription-manager repos --enable satellite-maintenance-6.12-for-rhel-8-x86_64-rpms
subscription-manager repos --enable satellite-6.12-for-rhel-8-x86_64-rpms

Then upgrade steps should be followed from Step 6 onward (https://access.redhat.com/documentation/en-us/red_hat_satellite/6.12/html/upgrading_and_updating_red_hat_satellite/upgrading_satellite).

The problem comes from missing module metadata in the Satellite Maintenance repository that prevents dnf from seeing that there is a newer version of rubygem-foreman_maintain available.

— Additional comment from on 2022-11-17T09:17:03Z

(In reply to Eric Helms from comment #1)

> The problem comes from missing module metadata in the Satellite Maintenance
> repository that prevents dnf from seeing that there is a newer version of
> rubygem-foreman_maintain available.

I don't this is entirely correct (but is still ultimately the cause for the issue at hand).

The problem is that rubygem-foreman_maintain (and other packages from the maintenance repo) are ALSO present in the Satellite and Capsule repositories.
Due to this fact, DNF considers these packages as part of a module (as Satellite and Capsule DO ship modular metadata, listing these packages) and modular packages cannot be updated by a non-modular package (as presented in the Maintenance repository).

This wasn't a problem for 6.11, as there the version in Satellite/Capsule and Maintenance was identical, but is now as we're telling people to only enable 6.12 Maintenance and let the tool do the rest for them.

Another (no idea if better) workaround would be to set the module_hotfixes flag on the Maintenance repository:

1. subscription-manager repo-override --repo=satellite-maintenance-6.12-for-rhel-8-x86_64-rpms --add=module_hotfixes:1

We can add modular metadata to the Maintenance repository, but that will mean users will have to explicitly enable that module too, which feels kinda wrong (another step to do manually) and will also require adjustments to the Maintenance code (for the self-upgrade feature).

IMHO the most correct way would be to explicitly mark the Maintenance related packages as "demodularized" in the Satellite/Capsule modules (https://github.com/fedora-modularity/libmodulemd/blob/2edaf37c260804824678c0c5ad9e592a19bafca6/yaml_specs/modulemd_stream_v2.yaml#L387-L405) and exclude them from being included in the "artifacts.rpms" list.

— Additional comment from on 2022-11-18T03:01:22Z

This seems like an urgent issue that should be documented with the work around in the upgrade documentation?

Also where was the QA?

By running:

subscription-manager repos --enable satellite-maintenance-6.12-for-rhel-8-x86_64-rpms
subscription-manager repos --enable satellite-6.12-for-rhel-8-x86_64-rpms

I was able to get past the issue on my personal setup.

Though I was trying other things before going to search and this is not exactly something anyone would expect.

— Additional comment from on 2022-11-23T13:48:56Z

(In reply to Eric Helms from comment #1)
> The workaround is to enable the satellite 6.12 repository (or Capsule
> repository if it's a Capsule) after enabling the maintenance repository:
>
> subscription-manager repos --enable
> satellite-maintenance-6.12-for-rhel-8-x86_64-rpms
> subscription-manager repos --enable satellite-6.12-for-rhel-8-x86_64-rpms
>
> Then upgrade steps should be followed from Step 6 onward
> (https://access.redhat.com/documentation/en-us/red_hat_satellite/6.12/html/
> upgrading_and_updating_red_hat_satellite/upgrading_satellite).
>
>
>
> The problem comes from missing module metadata in the Satellite Maintenance
> repository that prevents dnf from seeing that there is a newer version of
> rubygem-foreman_maintain available.

One thing I would like to note to be aware of with this work around – this enables the 6.12 server repository (or Capsule if upgrading a Capsule) and thus makes available package updates. If you enable the repositories, and run into issues as part of 'upgrade
check' you must ensure you do not yourself or have tooling that runs yum update or that you disable the 'satellite-6.12-for-rhel-8-x86_64-rpms' repository while the errors from upgrade check are being investigated.

— Additional comment from on 2022-11-25T09:35:22Z

While this bug is open, customers continue to get bad experience of the product due no mention of the bug in docs. There definately should be note at the docs until it's fixed, acccording to "customer first" attitude we should have. Doc should have link to this: https://access.redhat.com/solutions/6986053

— Additional comment from on 2022-11-25T12:25:00Z

Hi Ilkka,
Thank you for the heads-up and your email.
Known issues are docuemnted in the Release Notes document, but in case of this one the info hasn't propagate to docs. Apologies.

Evgeni, please could you provide the source info about the cause (what is causing the bug), consequence (what user experiences because of the bug) and workaround (how to workaround the issue) so that I can write a description that can be immediately added to Satellite Release Notes?

Thank you!

— Additional comment from on 2022-11-25T12:49:10Z

@Marie
How in the world is anyone trying to upgrade to 6.12 going to find anything in release notes? This needs to be in the main documentation or fixed ASAP. You can see from my comment above that this is horrible and shows a failure from Red Hat and Red Hat QA.

— Additional comment from on 2022-11-25T13:03:51Z

FailedQA.

@Satellite 6.11.4.2 Snap1

No satellite module at internal repo /pulp/repos/Sat6-CI/6_11_4_2-1_0/Satellite_6_11_with_RHEL8_Server/custom/Satellite_6_11_Composes/Satellite_6_11_RHEL8/ !!!
Without modularity information in the satellite repo I am not able to test de-modularization of maintenance packages

>>> everything is demodularize - snap templatization is failing to produce 6.11.4.2 el8 template

— Additional comment from on 2022-11-25T14:09:40Z

Hello,

The fix of procedures in product docs has been requested via a separate doc bug (https://bugzilla.redhat.com/show_bug.cgi?id=2148435). The bug is now assigned to one of our tech writers and a PR will be raised soon.

Feel free to watch the progress there.

Also, the docs team will update this engineering BZ accordingly when there is some progress.

Thank you!

— Additional comment from on 2022-11-25T14:57:30Z

The information about this know issue has been added to Satellite 6.12 RNs and is now available at RH Customer Portal:

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.12/html/release_notes/assembly_introducing-red-hat-satellite_sat6-release-notes#ref_known-issues_assembly_introducing-red-hat-satellite

— Additional comment from on 2022-11-25T15:13:42Z

Update of 6.11 docs is in progress (a PR already exists) - see updates in https://bugzilla.redhat.com/show_bug.cgi?id=2148435

A new BZ to update procedures in 6.12 docs has been raised> https://bugzilla.redhat.com/show_bug.cgi?id=2148466

— Additional comment from on 2022-11-29T08:50:06Z

I've been told my input is not required anymore, clearing needinfo

— Additional comment from on 2022-11-30T16:02:37Z

Copying my commentary from https://issues.redhat.com/browse/SAT-14019

There shouldn't have been any need to include libsolv (or libmodulemd) with Satellite in RHEL 8 - these dependencies were needed on RHEL 7 because the versions that exist there were too old, and it was safe because they were not core components of the system (yum didn't use them). The reverse is true with RHEL 8, these are core system components that shouldn't be touched, and we have no need to touch them because RHEL 8 provides them and keeps them up-to-date.

— Additional comment from on 2022-12-12T21:24:36Z

• • • Bug 2144057 has been marked as a duplicate of this bug. ***

— Additional comment from on 2023-02-02T14:22:37Z

(Note that my last comment was about the wrong issue, apologies. Mixed up my links.)

— Additional comment from on 2023-02-23T17:28:48Z

Eric,

Is there any action or work needed in Satellite 6.13 for this one?

If not, would should this go through QE (e.g. verification of 6.12-6.13 upgrade on stage or other env)?

— Additional comment from on 2023-02-28T01:06:14Z

I'm kinda worried because I'm following the steps from https://access.redhat.com/solutions/6986053 and it is not working for me...
[root@sat6 ~]# subscription-manager repos --enable satellite-6.12-for-rhel-8-x86_64-rpms
Repository 'satellite-6.12-for-rhel-8-x86_64-rpms' is enabled for this system.
[root@sat6 ~]# satellite-maintain upgrade list-versions
Checking for new version of satellite-maintain...

rubygem-foreman_maintain.noarch 1:1.1.9-1.el8sat satellite-maintenance-6.12-for-rhel-8-x86_64-rpms
rubygem-foreman_maintain.noarch 1:1.1.9-1.el8sat satellite-6.12-for-rhel-8-x86_64-rpms

Updating satellite-maintain package.

The satellite-maintain package successfully updated.
Re-run satellite-maintain with required options!
[root@sat6 ~]# satellite-maintain upgrade list-versions
Checking for new version of satellite-maintain...
Nothing to update, can't find new version of satellite-maintain.
6.11.z

— Additional comment from on 2023-02-28T01:07:25Z

[root@sat6 ~]# dnf history info 5

Updating Subscription Management repositories.
Reading Foreman protector configuration

• • • Excluded total: 0

Nothing excluded by foreman-protector!

Transaction ID : 5
Begin time : Mon 27 Feb 2023 07:58:55 PM EST
Begin rpmdb : 4074:35a662b36f48563cc2575ab471a0654897493bb8
End time : Mon 27 Feb 2023 07:58:57 PM EST (2 seconds)
End rpmdb : 4074:6ae4f55f24a4b0ac06d15069a5d5cb0dfda4a3d6
User : root <root>
Return-Code : Success
Releasever : 8
Command Line : -y --disableplugin=foreman-protector update rubygem-foreman_maintain
Comment :
Packages Altered:
Upgrade rubygem-foreman_maintain-1:1.1.9-1.el8sat.noarch @satellite-maintenance-6.12-for-rhel-8-x86_64-rpms
Upgraded rubygem-foreman_maintain-1:1.0.18-1.el8sat.noarch @@System

— Additional comment from on 2023-02-28T09:59:22Z

When we upgrade , the entire scenario is captured inside /var/lib/foreman-maintain/data.yml

If 6.11 upgrade or 6.11.z upgrade had failed at some point and you had completed it manually, Then the file /var/lib/foreman-maintain/data.yml will still not have recorded the completion details for that 6.11\6.11.z upgrade.

The only way to see the 6.12 in the list of upgradable versions would be :

A) Re-run the last upgrade via satellite-maintain ( which had failed earlier )

B) Or, Remove /var/lib/foreman-maintain/data.yml file and then check back on "satellite-maintain upgrade list-versions" output

This is only to be done if the upgrade was already and successfully finished via manual efforts , after the initial failure.

— Additional comment from on 2023-03-21T11:45:42Z

VERIFIED.

@Satellite 6.13.0 Snap15
rubygem-foreman_maintain-1.2.7-1.el8sat.noarch

by the following reproducers:

#1 INSTALLATION

1) Subscribe and enable required repositories (+ internal candidate repos)
2) Enable satellite:el8 module

1. dnf module enable satellite:el8
   ...
   Enabling module streams:
   pki-core 10.6
   pki-deps 10.6
   postgresql 12
   ruby 2.7
   satellite el8
   satellite-maintenance el8

>>> satellite-maintenance module gets enabled automatically as a dependency of satellite module

3) Install satellite rpms

1. dnf install satellite
   4) Run installer
2. satellite-installer -S satellite

#2 UPGRADE

1) Have a Satellite 6.12 subscribed with required repos enabled

2) Enable Satellite 6.13 Maintenance repo (internal candidate repo atm)

3) Check whether both product modules are available

1. dnf module list satellite satellite-maintenance
   ...
   Satellite Maintenance 6.13 RHEL8
   Name Stream Profiles Summary
   satellite-maintenance el8 Satellite Maintenance module

Red Hat Satellite 6.12 for RHEL 8 x86_64 (RPMs)
Name Stream Profiles Summary
satellite el8 [e] Satellite module

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

>>> there is now new satellite-maintenance module provided by 6.13 maintenance repository

4) Enable satellite-maintenance module in order to get updates
Note: this is required only once - during next upgrade this step won't needed (vs. enabling next satellite repo as the workaround was required during every upgrade)

1. dnf module enable satellite-maintenance:el8

5) Try to update maintenance rubygem

1. satellite-maintain upgrade list-versions
   Checking for new version of satellite-maintain...

rubygem-foreman_maintain.noarch 1:1.2.7-1.el8sat Satellite_Engineering_Satellite_6_13_Composes_Satellite_Maintenance_6_13_RHEL8

Updating satellite-maintain package.

1. satellite-maintain upgrade list-versions
   Checking for new version of satellite-maintain...
   Nothing to update, can't find new version of satellite-maintain.
   6.12.z
   6.13

>>> rubygem-foreman_maintain is successfully updated from next version repo