What were you trying to do that didn't work?

Satellite (starting with 6.17) and upstream Foreman enables Candlepin's multiple-environment feature, where a host (consumer) can be registered to multiple Candlepin environments.

I was trying to dnf install / info on a package, and got this:

# dnf info jq
Updating Subscription Management repositories.
1 local certificate has been deleted.
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                         1.4 kB/s |  73  B     00:00    
Errors during downloading metadata for repository 'rhel-9-for-x86_64-baseos-rpms':
  - Status code: 403 for https://centos9-katello-devel.fedora.example.com/pulp/content/Default_Organization/Library/content/dist/rhel9/9/x86_64/baseos/os/repodata/repomd.xml (IP: 192.168.122.128)
Error: Failed to download metadata for repo 'rhel-9-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

The issue only occurs when (a) the host (consumer) is assigned multiple Candlepin environments; and (b) the 'Library' environment is not ordered first.

For example, set environments as follows on a Satellite-registered host with subscription-manager environments --set:

Library,Library/Eppel <-- works

Library/Eppel,Library <-- doesn't work

Library/Misc,Library/Eppel <-- works

Library/Eppel,Library/Misc <-- works

What is the impact of this issue to you?

Satellite customers who use the new multiple-content-views feature may not be able to access repos in Library environment.

Please provide the package NVR for which the bug is seen:

subscription management server: 4.4.16-1
subscription management rules: 5.44
subscription-manager: 1.29.40-1.el9

How reproducible is this bug?:

theoretically 100%, if you follow the reproducer steps

Steps to reproduce

You can test this on Satellite Stream (current) or upstream Foreman/Katello nightly. The allow_multiple_content_views setting must be turned on.

1. Sync RHEL9 BaseOS and AppStream
2. Sync at least one other repo
3. Create a content view with that other repo, and publish it (let's call it cv1)
4. Turn on the allow_multiple_content_views setting
5. Register a host and assign it to environments Library/cv1, Library
6. On the host, dnf info jq

Expected results

info is displayed

Actual results

Error: Failed to download metadata for repo 'rhel-9-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Pulp logs on Satellite (journalctl -u pulp ) show

None]: pulp_certguard.app.models:WARNING: Path /Default_Organization/Library/content/dist/rhel9/9/x86_64/baseos/os/repodata/repomd.xml is not allowed in client cert

even though the same path is shown on the host cert in /etc/pki/entitlement. But Pulp is just asking subscription-manager about the paths: https://github.com/pulp/pulpcore/blob/main/pulp_certguard/app/models.py#L184

The error occurs regardless of the host's releasever setting.