Description of problem:

If the satellite\capsule servers are missing artifacts related to both rpm and docker type repos, Then The Repair API, when executed, may not be able to repair all the rpm data and stop it's action if it fails to process any docker manifest-related artifacts.

Version-Release number of selected component (if applicable):

satellite-6.13.0-6.el8sat.noarch
katello-4.7.0-1.el8sat.noarch
python39-pulpcore-3.21.5-1.1.el8pc.noarch
python39-pulp-rpm-3.18.11-1.el8pc.noarch

How reproducible:

Always

Steps to Reproduce:
1. Complete The reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=2173671

2. Enable Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs)

3. Change the download policy to immediate and sync the repo

4. Ensure that we have two repos enabled now in satellite, yum as well as docker type

5. Delete any artifacts related to the satellite client repo

6. Run the Repair API

Actual results:

Step 3:

Added Rpms: 16, Errata: 3
Total steps: 65/65
--------------------------------
Associating Content: 20/20
Downloading Artifacts: 16/16
Downloading Metadata Files: 9/9
Parsed Advisories: 3/3
Parsed Comps: 1/1
Parsed Packages: 16/16
Skipping Packages: 0/0
Un-Associating Content: 0/0

Step 4:

1. hammer repository list --fields "Id,Name,Content Type"
   --|-------------------------------------------------|------------
   ID | NAME | CONTENT TYPE
   --|-------------------------------------------------|------------
   2 | Red Hat Satellite Client 6 for RHEL 8 x86_64 RPMs | yum
   1 | rhel8/toolbox | docker
   --|-------------------------------------------------|------------

Step 5:

1. echo "COPY (select base_path from core_distribution where base_path like '%client%') TO STDOUT;" | su - postgres -c "psql pulpcore"
   RedHat/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os

1. echo "COPY(select cpa.relative_path,ca.file from core_distribution as core_d RIGHT JOIN core_publication as cp ON core_d.publication_id=cp.pulp_id LEFT JOIN core_publishedartifact as cpa ON cp.pulp_id=cpa.publication_id LEFT JOIN core_contentartifact as core_ca ON core_ca.pulp_id=cpa.content_artifact_id LEFT JOIN core_artifact as ca ON ca.pulp_id=core_ca.artifact_id where core_d.base_path = 'RedHat/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os' order by core_d.base_path) TO STDOUT WITH CSV" | su - postgres -c "psql pulpcore" > /tmp/filename_to_artifact_mapping.txt

1. for map in $(cat /tmp/filename_to_artifact_mapping.txt)
   do
   FILENAME=$(echo $map | awk -F',' ' {print $1}')
   FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)"
   if [ $FULLPATH == "/var/lib/pulp/media/" ]
   then
   STAT="Not-Downloaded"
   else
   STAT=$(test -e $FULLPATH && echo "PRESENT" || echo "ABSENT")
   fi
   echo "$FILENAME,$STAT"
   done | column -s, -t
   
   Packages/f/foreman_ygg_worker-0.1.1-2.el8sat.x86_64.rpm PRESENT
   Packages/g/gofer-2.12.5-7.el8sat.noarch.rpm PRESENT
   Packages/g/gofer-2.12.5-8.el8sat.noarch.rpm PRESENT
   Packages/k/katello-agent-3.5.7-3.el8sat.noarch.rpm PRESENT
   Packages/k/katello-host-tools-3.5.7-3.el8sat.noarch.rpm PRESENT
   Packages/k/katello-host-tools-tracer-3.5.7-3.el8sat.noarch.rpm PRESENT
   Packages/k/katello-pull-transport-migrate-1.0.2-1.el8sat.noarch.rpm PRESENT
   Packages/p/puppet-agent-7.16.0-2.el8sat.x86_64.rpm PRESENT
   Packages/p/python3-gofer-2.12.5-7.el8sat.noarch.rpm PRESENT
   Packages/p/python3-gofer-2.12.5-8.el8sat.noarch.rpm PRESENT
   Packages/p/python3-gofer-proton-2.12.5-7.el8sat.noarch.rpm PRESENT
   Packages/p/python3-gofer-proton-2.12.5-8.el8sat.noarch.rpm PRESENT
   Packages/p/python3-qpid-proton-0.33.0-4.el8.x86_64.rpm PRESENT
   Packages/q/qpid-proton-c-0.33.0-4.el8.x86_64.rpm PRESENT
   Packages/r/rubygem-foreman_scap_client-0.5.0-1.el8sat.noarch.rpm PRESENT
   Packages/y/yggdrasil-0.2.0-2.el8sat.x86_64.rpm PRESENT
   repodata/2ec7ea4c19077fc0f495944e5ed4a8bb3dc9af45bfe6634dbb7aabc2cb41fb6d-updateinfo.xml.gz PRESENT
   repodata/37e6338d623013fd1285fd13033596e2e36c55f0db8b6635f8422a079fb98568-primary.xml.gz PRESENT
   repodata/4badd0660333ce1828baa560d2208b1439e969de67d9848301894614310f221a-filelists.sqlite.bz2 PRESENT
   repodata/77895819bddc7dcd7cb8ad334cfe33aee2f00c0fd03ab663914ce280944e4ff2-comps.xml PRESENT
   repodata/9b0bd210bfc9887bf17a2bf26e986285a9ab3190cec96b5ec5972df56c546713-filelists.xml.gz PRESENT
   repodata/c4cc5c111ae7df9225eea3f1ac0bb6dd44706aeb643b9b89495d1bb16cdf0fd9-other.xml.gz PRESENT
   repodata/e735df44940193e25ea4e467630ef8a4e94e55d4b4d37e320d463dc3f5ca1669-other.sqlite.bz2 PRESENT
   repodata/fdddabeef43c982760a1c94724e98f9314974fe7a0b67e13c12af5726e48025f-primary.sqlite.bz2 PRESENT
   repodata/repomd.xml PRESENT
   
   
   # for map in $(cat /tmp/filename_to_artifact_mapping.txt)
   do
   FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)"
   if [ $FULLPATH == "/var/lib/pulp/media/" ]
   then
   echo ''
   else
   rm -f $FULLPATH
   fi
   done
   
   # for map in $(cat /tmp/filename_to_artifact_mapping.txt)
   do
   FILENAME=$(echo $map | awk -F',' '{print $1}

   ')
   FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)"
   if [ $FULLPATH == "/var/lib/pulp/media/" ]
   then
   STAT="Not-Downloaded"
   else
   STAT=$(test -e $FULLPATH && echo "PRESENT" || echo "ABSENT")
   fi
   echo "$FILENAME,$STAT"
   done | column -s, -t

Packages/f/foreman_ygg_worker-0.1.1-2.el8sat.x86_64.rpm ABSENT
Packages/g/gofer-2.12.5-7.el8sat.noarch.rpm ABSENT
Packages/g/gofer-2.12.5-8.el8sat.noarch.rpm ABSENT
Packages/k/katello-agent-3.5.7-3.el8sat.noarch.rpm ABSENT
Packages/k/katello-host-tools-3.5.7-3.el8sat.noarch.rpm ABSENT
Packages/k/katello-host-tools-tracer-3.5.7-3.el8sat.noarch.rpm ABSENT
Packages/k/katello-pull-transport-migrate-1.0.2-1.el8sat.noarch.rpm ABSENT
Packages/p/puppet-agent-7.16.0-2.el8sat.x86_64.rpm ABSENT
Packages/p/python3-gofer-2.12.5-7.el8sat.noarch.rpm ABSENT
Packages/p/python3-gofer-2.12.5-8.el8sat.noarch.rpm ABSENT
Packages/p/python3-gofer-proton-2.12.5-7.el8sat.noarch.rpm ABSENT
Packages/p/python3-gofer-proton-2.12.5-8.el8sat.noarch.rpm ABSENT
Packages/p/python3-qpid-proton-0.33.0-4.el8.x86_64.rpm ABSENT
Packages/q/qpid-proton-c-0.33.0-4.el8.x86_64.rpm ABSENT
Packages/r/rubygem-foreman_scap_client-0.5.0-1.el8sat.noarch.rpm ABSENT
Packages/y/yggdrasil-0.2.0-2.el8sat.x86_64.rpm ABSENT
repodata/2ec7ea4c19077fc0f495944e5ed4a8bb3dc9af45bfe6634dbb7aabc2cb41fb6d-updateinfo.xml.gz ABSENT
repodata/37e6338d623013fd1285fd13033596e2e36c55f0db8b6635f8422a079fb98568-primary.xml.gz ABSENT
repodata/4badd0660333ce1828baa560d2208b1439e969de67d9848301894614310f221a-filelists.sqlite.bz2 ABSENT
repodata/77895819bddc7dcd7cb8ad334cfe33aee2f00c0fd03ab663914ce280944e4ff2-comps.xml ABSENT
repodata/9b0bd210bfc9887bf17a2bf26e986285a9ab3190cec96b5ec5972df56c546713-filelists.xml.gz ABSENT
repodata/c4cc5c111ae7df9225eea3f1ac0bb6dd44706aeb643b9b89495d1bb16cdf0fd9-other.xml.gz ABSENT
repodata/e735df44940193e25ea4e467630ef8a4e94e55d4b4d37e320d463dc3f5ca1669-other.sqlite.bz2 ABSENT
repodata/fdddabeef43c982760a1c94724e98f9314974fe7a0b67e13c12af5726e48025f-primary.sqlite.bz2 ABSENT
repodata/repomd.xml ABSENT

1. file /var/lib/pulp/media/artifact//
   /var/lib/pulp/media/artifact/1c/b61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae: gzip compressed data, from Unix, original size 167
   /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text
   /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808
   /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824
   /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864
   /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/95/a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f: gzip compressed data, from Unix, original size 125
   /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648
   /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/c0/cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898: gzip compressed data, from Unix, original size 59
   /var/lib/pulp/media/artifact/cb/920667676480deb7ebd62198e571e416462118d58b9c280b413fec31745e4e: XML 1.0 document, ASCII text
   /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/ef/3e20691954c3d1318ec3071a982da339f4ed76967ded668b795c9e070aaab6: gzip compressed data, from Unix, original size 121

Step 6:

1. curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 67 100 67 0 0 683 0 -::- -::- -::- 683
   {
   "task": "/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/"
   }
2. curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X GET https://`hostname -f`/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/ | json_reformat
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 2860 100 2860 0 0 42058 0 -::- -::- -::- 42058
   {
   "pulp_href": "/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/",
   "pulp_created": "2023-02-27T17:00:38.933371Z",
   "state": "failed",
   "name": "pulpcore.app.tasks.repository.repair_all_artifacts",
   "logging_cid": "3997124f3767438e86884df2e6d3519d",
   "started_at": "2023-02-27T17:00:38.970659Z",
   "finished_at": "2023-02-27T17:00:40.625673Z",
   "error": { "traceback": " File \"/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 452, in _perform_task\n result = func(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 192, in repair_all_artifacts\n loop.run_until_complete(_repair_artifacts_for_content(verify_checksums=verify_checksums))\n File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 148, in _repair_artifacts_for_content\n await asyncio.gather(*done) # Clean up tasks\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 77, in _repair_ca\n dl_result = await downloader.run()\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 273, in run\n return await download_wrapper()\n File \"/usr/lib/python3.9/site-packages/backoff/_async.py\", line 151, in retry\n ret = await target(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 258, in download_wrapper\n return await self._run(extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 81, in _run\n return await self._run(handle_401=False, extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 66, in _run\n response.raise_for_status()\n File \"/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py\", line 1004, in raise_for_status\n raise ClientResponseError(\n", "description": "404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677520840~hmac=4c1a924c0c8b5af741a5025eaa730f8b5fca99f539f5004d65aeb6bc136c3db4')" }

   ,
   "worker": "/pulp/api/v3/workers/60781c7e-a9fd-4386-8ba6-6c7d4785a9a1/",
   "parent_task": null,
   "child_tasks": [

],
"task_group": null,
"progress_reports": [

{ "message": "Repair corrupted units", "code": "repair.repaired", "state": "failed", "total": null, "done": 0, "suffix": null }

,

{ "message": "Identify corrupted units", "code": "repair.corrupted", "state": "failed", "total": null, "done": 0, "suffix": null }

,

{ "message": "Identify missing units", "code": "repair.missing", "state": "failed", "total": null, "done": 6, "suffix": null }

],
"created_resources": [

],
"reserved_resources_record": [

]
}

1. file /var/lib/pulp/media/artifact//
   /var/lib/pulp/media/artifact/1c/b61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae: gzip compressed data, from Unix, original size 167
   /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text
   /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808
   /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824
   /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864
   /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/95/a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f: gzip compressed data, from Unix, original size 125
   /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648
   /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/c0/cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898: gzip compressed data, from Unix, original size 59
   /var/lib/pulp/media/artifact/cb/920667676480deb7ebd62198e571e416462118d58b9c280b413fec31745e4e: XML 1.0 document, ASCII text
   /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/ef/3e20691954c3d1318ec3071a982da339f4ed76967ded668b795c9e070aaab6: gzip compressed data, from Unix, original size 121

Expected results:

The Repair API should be able to skip the failed units and proceed fixing the other RPM-type units that It can.

Additional info:

• This is not a huge blocker for satellite as satellite ( via katello ) offers an easy way to run repair API per repo .

• But when it comes to Capsules, That is not the case and hence the Repair action needs to be executed for all repos and that will fails at the very beginning ( under the specific condition as mentioned earlier )