Description of problem:

Repair API of Pulp3 ( Verify Content Checksum of Katello ) cannot get back\fix docker manifests

Version-Release number of selected component (if applicable):

satellite-6.13.0-6.el8sat.noarch
katello-4.7.0-1.el8sat.noarch
python39-pulpcore-3.21.5-1.1.el8pc.noarch
python39-pulp-rpm-3.18.11-1.el8pc.noarch

How reproducible:
Always

Steps to Reproduce:

1. Install a Satellite 6.13 and import a manifest with some valid subs
2. Use the repo_discovery feature to discover the "rhel8/toolbox" repo and create it inside the Containers product
3. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox and add "8.7-5" in the "Include Tags" fields and sync the repo. ( It will only sync that tag now )
4. Delete the artifacts

1. rm -rf /var/lib/pulp/media/artifact/*

5. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox --> Select Action dropdown --> Sync the repo.

6. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox --> Select Action dropdown --> Verify Content Checksum

Or else just invoke the Repair API

1. curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat

Actual results:

Step 3: Sync status from Sat UI:

Added Docker tags: 1
Total steps: 32/32
--------------------------------
Associating Content: 18/18
Downloading Artifacts: 12/12
Downloading tag list: 1/1
Processing Tags: 1/1
Un-Associating Content: 0/0

1. du -sh /var/lib/pulp/media/artifact/
   556M /var/lib/pulp/media/artifact/

1. file /var/lib/pulp/media/artifact//
   /var/lib/pulp/media/artifact/01/65232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276: ASCII text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text
   /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808
   /var/lib/pulp/media/artifact/62/9545c824752b6d75f7f22a73b6c7fcba9413b6017f2f67f58494a958bcd4ce: ASCII text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824
   /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864
   /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/7f/060f344d579611663e84ea94d89d9372d7431e3e56e11a19e704207d5576e9: ASCII text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/9a/394c72313c9bcac63dc8980018582d90ece7ae65d03940e6d7e9b8d4cb884d: gzip compressed data, original size 264029696
   /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648
   /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/b1/45dc70f6c036069d549762f39055162cac12b414e9e502fd0d4b3331a0f5eb: gzip compressed data, original size 171049472
   /var/lib/pulp/media/artifact/b9/2727ef7443d73bb77323e5a969bf02187286fccf53930e83dc82be60a36529: gzip compressed data, original size 214139392
   /var/lib/pulp/media/artifact/c6/25ce0354e96c43d4bfca9cb0a3ed8373b0707980f152582f731e61808df9bd: ASCII text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/d4/d59e4eb38ff5eda9a9f6255575abc64501adb78153f5788bf580b1034d4f8d: gzip compressed data, original size 186984960
   /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators

Step 4:

1. rm -rf /var/lib/pulp/media/artifact/*

1. du -sh /var/lib/pulp/media/artifact/
   0 /var/lib/pulp/media/artifact/

Step 5:

Sync fails but able to get some data in pulp:

1. file /var/lib/pulp/media/artifact//
   /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text

Step 6:

1. curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 67 100 67 0 0 881 0 -::- -::- -::- 881
   {
   "task": "/pulp/api/v3/tasks/bff6cd3c-c7cb-43c1-a717-ec68512e10fa/"
   }

1. curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X GET https://`hostname -f`/pulp/api/v3/tasks/bff6cd3c-c7cb-43c1-a717-ec68512e10fa/ | json_reformat
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 2860 100 2860 0 0 46885 0 -::- -::- -::- 46885
   {
   "pulp_href": "/pulp/api/v3/tasks/bff6cd3c-c7cb-43c1-a717-ec68512e10fa/",
   "pulp_created": "2023-02-27T15:30:24.465374Z",
   "state": "failed",
   "name": "pulpcore.app.tasks.repository.repair_all_artifacts",
   "logging_cid": "15f07cd0ac6146968f91e6b229189b3f",
   "started_at": "2023-02-27T15:30:24.508613Z",
   "finished_at": "2023-02-27T15:30:26.289194Z",
   "error": { "traceback": " File \"/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 452, in _perform_task\n result = func(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 192, in repair_all_artifacts\n loop.run_until_complete(_repair_artifacts_for_content(verify_checksums=verify_checksums))\n File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 148, in _repair_artifacts_for_content\n await asyncio.gather(*done) # Clean up tasks\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 77, in _repair_ca\n dl_result = await downloader.run()\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 273, in run\n return await download_wrapper()\n File \"/usr/lib/python3.9/site-packages/backoff/_async.py\", line 151, in retry\n ret = await target(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 258, in download_wrapper\n return await self._run(extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 81, in _run\n return await self._run(handle_401=False, extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 66, in _run\n response.raise_for_status()\n File \"/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py\", line 1004, in raise_for_status\n raise ClientResponseError(\n", "description": "404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d')" }

   ,
   "worker": "/pulp/api/v3/workers/854914bf-6e9d-4bd9-8c84-d13557f44e82/",
   "parent_task": null,
   "child_tasks": [

],
"task_group": null,
"progress_reports": [

{ "message": "Repair corrupted units", "code": "repair.repaired", "state": "failed", "total": null, "done": 0, "suffix": null }

,

{ "message": "Identify corrupted units", "code": "repair.corrupted", "state": "failed", "total": null, "done": 0, "suffix": null }

,

{ "message": "Identify missing units", "code": "repair.missing", "state": "failed", "total": null, "done": 6, "suffix": null }

],
"created_resources": [

],
"reserved_resources_record": [

]
}

Trace from logs:

Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: pulp_container.app.downloaders:INFO: Updating bearer token
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?auth=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d'))
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?auth=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d'))
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: pulpcore.tasking.pulpcore_worker:INFO: Task bff6cd3c-c7cb-43c1-a717-ec68512e10fa failed (404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?auth=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d'))
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: result = func(*args, **kwargs)
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py", line 192, in repair_all_artifacts
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: loop.run_until_complete(_repair_artifacts_for_content(verify_checksums=verify_checksums))
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return future.result()
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py", line 148, in _repair_artifacts_for_content
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: await asyncio.gather(*done) # Clean up tasks
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py", line 77, in _repair_ca
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: dl_result = await downloader.run()
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 273, in run
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return await download_wrapper()
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: ret = await target(*args, **kwargs)
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 258, in download_wrapper
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return await self._run(extra_data=extra_data)
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py", line 81, in _run
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return await self._run(handle_401=False, extra_data=extra_data)
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py", line 66, in _run
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: response.raise_for_status()
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: raise ClientResponseError(

Filesystem state:

1. file /var/lib/pulp/media/artifact//
   /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text
   /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808
   /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824
   /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864
   /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648
   /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators
   /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators

Expected results:

The Repair API should be able to gracefully get back all the syncable data ( even for container\docker type repos )

Additional info:

• Reproducer details shared in the next private comment

• It has been noticed that, if this API fails to fix something, Then it immediately ends the task instead of checking for the other fixable objects\units and trying to fix\redownload them.