Description of problem:

Sequence:

• A file-type repo exists with no upstream URL specified.
• A file was uploaded to the repo manually
• The repo was made part of a CV and CV was republished

If we remove the artifact related to that file from the filesystem, then The only way to get back the file is by re-uploading it in the repo manually.

But even if the reupload would be successful, The file would still not be accessible from it's Published path

Version-Release number of selected component (if applicable):

Satellite 6.14

1. rpm -q satellite katello python39-pulpcore
   satellite-6.14.0-3.el8sat.noarch
   katello-4.9.0-1.el8sat.noarch
   python39-pulpcore-3.22.7-1.el8pc.noarch

How reproducible:

Always

Steps to Reproduce:
1. Install Satellite 6.14.0 with Org Name RedHat

2. Create a product called "Custom"

3. Create a file type repo called "FILE_NO_URL" in that product and that should not have any upstream URLs mentioned and it should be unprotected.

4. Upload a file called "rake_options" in that repo from UI or hammer

5. Create a CV called "TestCV", add that FILE_NO_URL repo in the CV and Publish a new version.

6. Do a curl on "http://satellite614.lab.example.com/pulp/content/RedHat/Library/custom/Custom/FILE_NO_URL/rake_options"

"http://satellite614.lab.example.com/pulp/content/RedHat/Library/custom/Custom/FILE_NO_URL/PULP_MANIFEST"

"http://satellite614.lab.example.com/pulp/content/RedHat/Library/TestCV/custom/Custom/FILE_NO_URL/rake_options"

"http://satellite614.lab.example.com/pulp/content/RedHat/Library/TestCV/custom/Custom/FILE_NO_URL/PULP_MANIFEST"

7. On satellite, "rm -rf /var/lib/pulp/media/artifact/*"

8. Repeat Step 6

9. Manually reupload the file rake_options back to the repo and perform a CV version republish as well.

10. Repeat Step 6

11. Delete the file from the repo ( using hammer or Sat UI )

12. Consider Orphan Protection Timeout and then run an orphan cleanup.

13. Repeat Step 9

14. Repeat Step 6

Actual results:

No errors on any other steps except, Step 8 , 10 and 14

The PULP_MANIFEST would be present and can be fetched but the file rake_options is always missing.

Jul 28 21:04:00 satellite614 pulpcore-content[1851]: [28/Jul/2023:15:34:00 +0000] "GET /pulp/content/RedHat/Library/custom/Custom/FILE_NO_URL/ HTTP/1.1" 200 720 "https://satellite614.lab.example.com/products/389/repositories/108" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: [2023-07-28 15:34:02 +0000] [1844] [ERROR] Error handling request
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: Traceback (most recent call last):
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: resp = await request_handler(request)
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib64/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: resp = await handler(request)
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: return await handler(request)
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: return await handler(request)
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 339, in cached_function
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: response = await self.make_entry(
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 378, in make_entry
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: response = await handler(*args, **kwargs)
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 243, in stream_content
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: return await self._match_and_stream(path, request)
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 601, in _match_and_stream
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: return await self._serve_content_artifact(ca, headers, request)
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 847, in _serve_content_artifact
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: raise Exception(_("Expected path '{}' is not found").format(path))
Jul 28 21:04:02 satellite614 pulpcore-content[1844]: Exception: Expected path '/var/lib/pulp/media/artifact/92/5f04066d1649d1882cfd72db4514bb55154e1d8248452caf077cf7e8e3e655' is not found

Expected results:

If we are uploading back the file, It should be available in the repo.

This would affect https://bugzilla.redhat.com/show_bug.cgi?id=2227276 , because if users have restored their satellite with no pulp data and want to recover the same, They can recover everything else but not the file type repos that is manually managed.

Additional info:

I could be wrong but The only two probable ways to fix this would be:

A) Keep the file in an HTTP-exposed location, create a PULP_MANIFEST, use that location to define an upstream URL in the repo and then run a Repair Action ( i.e. Validate Content Checksum )

B) Or Completly delete the repo from everywhere in satellite, Cleanup orphaned contents and then recreate the repo with the files reuploaded in the repo

And then again add back the repo in CV and republish the CV

This is perhaps NOT a regression and it exists in older versions but I never tried this approach before until I was testing for BZ# 2227276 .