+++ This bug was initially created as a clone of Bug #2208310 +++

Description of problem:
Remove orphans task fails on Capsule servers. It executes successfully on the Satellite server itself.

Here are task details:

1. hammer task info --id 19c79c4f-eaea-4739-bea0-a8aec0f595c0
   ~~~
   ID: 19c79c4f-eaea-4739-bea0-a8aec0f595c0
   Action: Remove orphans
   State: paused
   Result: error
   Started at: 2023/05/14 20:00:28
   Ended at:
   Duration: 61876.254165
   Owner: foreman_admin
   Task errors: undefined method `pulp3_api' for nil:NilClass

Details:
Errors: undefined method `pulp3_api' for nil:NilClass
Action: Actions::Pulp3::OrphanCleanup::DeleteOrphanAlternateContentSources
Exception: NoMethodError: undefined method `pulp3_api' for nil:NilClass
Stack: 2023-05-14T22:00:58 [E|bac|] undefined method `pulp3_api' for nil:NilClass (NoMethodError)

Version-Release number of selected component (if applicable):
Satellite 6.13

How reproducible:
N/A

— Additional comment from on 2023-05-18T14:47:28Z

This looks like something that needs to be handled in katello, not in tasks as an engine.

— Additional comment from on 2023-05-18T14:55:10Z

So far wasn't able to reproduce this myself.

More details from customer's environment:

• This isn't fresh Satellite 6.13 install. Satellite was upgraded from 6.12. Capsules likely too, will confirm.
• Error can be reproduced with: # foreman-rake katello:delete_orphaned_content
• Satellite sosreport: https://access.redhat.com/support/cases/#/case/03511962/discussion?attachmentId=a096R000034289XQAQ
• task-export: https://access.redhat.com/support/cases/#/case/03511962/discussion?attachmentId=a096R000034289XQAQ

— Additional comment from on 2023-05-19T14:43:41Z

More information:

• Satellite and all Capsules were upgraded from 6.12 to 6.13.

• No ACS defined:

[root@sat-isp-to-p01 ~]# hammer alternate-content-source list
--|----|----
ID | NAME | TYPE
--|----|----

• Download policy for Capsules is 'On Demand'.

• Capsule features:

[root@sat-isp-to-p01 ~]# hammer capsule info --id 2
Id: 2
Name: cap-isp-to-p01.sede.corp.sanpaoloimi.com
URL: https://cap-isp-to-p01.sede.corp.sanpaoloimi.com:9090
Host_count: 0
Features:
1) Name: Pulpcore
2) Name: Templates
3) Name: TFTP
4) Name: Logs
5) Name: Registration
Locations:
ISP
Organizations:
ISP
Created at: 2023/05/02 08:13:24
Updated at: 2023/05/02 08:13:24

• Code the task is failing on:

—
/usr/share/gems/gems/katello-4.7.0.23/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:82
file_acs_api = ::Katello::Pulp3::Repository.api(smart_proxy, 'file').alternate_content_source_api <------

/usr/share/gems/gems/katello-4.7.0.23/app/services/katello/pulp3/repository.rb:42
def self.api(smart_proxy, repository_type_label)
repo_type = RepositoryTypeManager.enabled_repository_types[repository_type_label] <-------- repository_type_label = 'file'
repo_type.pulp3_api(smart_proxy)
end
—

• Possibly related . Customer disabled a bunch of features when they first installed their Satellite and Capsules (fresh 6.12):

satellite-installer --scenario satellite \
--foreman-initial-organization "ISP" \
--foreman-initial-location "ISP" \
--foreman-initial-admin-password "***" \
--tuning "large" \
--certs-server-cert "/root/certs/sat-isp-to-p01/sat-isp-to-p01_cer.pem" \
--certs-server-key "/root/certs/sat-isp-to-p01/sat-isp-to-p01_key.pem" \
--certs-server-ca-cert "/root/certs/ISGS_root_CA_cer.pem" \
--no-enable-foreman-plugin-ansible \
--no-enable-foreman-plugin-remote-execution \
--no-enable-foreman-plugin-azure \
--no-enable-foreman-plugin-openscap \
--no-enable-foreman-plugin-rh-cloud \
--no-enable-foreman-plugin-leapp \
--no-enable-foreman-cli-ansible \
--no-enable-foreman-cli-remote-execution \
--no-enable-foreman-cli-azure \
--no-enable-foreman-proxy-plugin-ansible \
--no-enable-foreman-proxy-plugin-discovery \
--no-enable-foreman-proxy-plugin-openscap \
--no-enable-foreman-proxy-plugin-remote-execution-script \
--foreman-proxy-content-enable-ansible false \
--foreman-proxy-content-enable-file false \
--foreman-proxy-content-enable-docker false

satellite-installer \
--scenario capsule\
--certs-tar-file "/root/cap-isp-to-p01.sede.corp.sanpaoloimi.com-certs.tar"\
--foreman-proxy-register-in-foreman "true"\
--foreman-proxy-foreman-base-url "https://sat-isp-to-p01.sede.corp.sanpaoloimi.com"\
--foreman-proxy-trusted-hosts "sat-isp-to-p01.sede.corp.sanpaoloimi.com"\
--foreman-proxy-trusted-hosts "cap-isp-to-p01.sede.corp.sanpaoloimi.com"\
--foreman-proxy-oauth-consumer-key "***"\
--foreman-proxy-oauth-consumer-secret "***" \
--tuning "large" \
--no-enable-foreman-proxy-plugin-remote-execution-script \
--foreman-proxy-tftp true \
--foreman-proxy-content-enable-ansible false \
--foreman-proxy-content-enable-file false \
--foreman-proxy-content-enable-docker false

— Additional comment from on 2023-06-01T14:24:06Z

Created redmine issue https://projects.theforeman.org/issues/36461 from this bug

— Additional comment from on 2023-06-14T16:03:15Z

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36461 has been resolved.

— Additional comment from on 2023-11-01T16:02:25Z

Fix in snap 35.

— Additional comment from on 2024-01-23T21:23:33Z

Reproduced in 6.13.6, verified in 6.15.0 snap 7.0 - the Remove orphans task succeeded when the file content was disabled on the external capsule and mocked in smart_proxy_mirror_repository.rb.

QE Tracker for https://issues.redhat.com/browse/SAT-23587
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2266149
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2208310