Description of problem:

When using download on-demand, if an artifact upstream changed between the moment the sync happened and the first time a user requests it, pulpcore-content will download the new file (which won't have the same checksum) and deliver it to the user without verifying if it is a match. While saving the downloaded file, a checksum verification is done and the artifact is not saved (because of mismatching checksum), but for the final user's eyes nothing wrong happened. Regardless of no errors being shown to the user, they receive a bad artifact.

Version-Release number of selected component (if applicable):

Observed in the wild on Satellite 6.11.
Reproduced internally on 6.11 and 6.13 (didn't test 6.12 and 6.14)

How reproducible:

Always when circumstances met.

Steps to Reproduce:

1. Create a local repository from where pulp will sync. Ensure you will be able to make changes on the repository content.

Example of a small repository, easy to create on localhost:

~~~
mkdir /var/www/html/pub/repo
cd /var/www/html/pub/repo
curl -O https://repos.fedorapeople.org/pulp/pulp/demo_repos/zoo/bear-4.1-1.noarch.rpm
createrepo .
~~~

2. On Satellite, create a product and a repository that will sync from http://satellite.example.com/pub/repo/ and sync it:

~~~
hammer product create --name TestProduct --organization-id 1
hammer repository create --url http://localhost/pub/repo/ --name ChangingRepo --product TestProduct --mirroring-policy mirror_complete --organization-id 1 --download-policy on_demand --content-type yum
hammer repository synchronize --name ChangingRepo --product TestProduct --organization-id 1
~~~

3. Change the content of the rpmfile bear-4.1-1.noarch.rpm from the upstream (on localhost) repository. Ensure the checksum of the file changed!!

~~~
$ sha256sum /var/www/html/pub/repo/bear-4.1-1.noarch.rpm
7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3 /var/www/html/pub/repo/bear-4.1-1.noarch.rpm <=== original checksum

$ echo "CORRUPTED" >> /var/www/html/pub/repo/bear-4.1-1.noarch.rpm <=== forcing a corruption of the file

$ sha256sum /var/www/html/pub/repo/bear-4.1-1.noarch.rpm
1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e /var/www/html/pub/repo/bear-4.1-1.noarch.rpm <==== corrupted checksum
~~~

4. Try downloading the file from the repository created on Satellite:

1. hammer repository info --name ChangingRepo --product TestProduct --organization-id 1 --fields "Published At"
   Published At: https://satellite.example.com/pulp/content/DefaultOrganization/Library/custom/TestProduct/ChangingRepo/

1. curl -O https://satellite.example.com/pulp/content/DefaultOrganization/Library/custom/TestProduct/ChangingRepo/bear-4.1-1.noarch.rpm
2. echo $? <=== note that command returns no error

1. sha256sum bear-4.1-1.noarch.rpm <====== checksum is something else

Actual results:

Client receives a bad file and no errors.

Pulpcore-content logs errors when saving the artifact:

~~~
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: Giving up download_wrapper(...) after 5 tries (pulpcore.exceptions.validation.DigestValidationError: A file located at the url http://localhost/pub/repo/bear-4.1-1.noarch.rpm failed validation due to checksum. Expected '7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3', Actual '1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e')
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (pulpcore.exceptions.validation.DigestValidationError: A file located at the url http://localhost/pub/repo/bear-4.1-1.noarch.rpm failed validation due to checksum. Expected '7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3', Actual '1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e')
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: [2024-01-31 16:10:08 +0000] [1141] [ERROR] Error handling request
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: Traceback (most recent call last):
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: resp = await request_handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib64/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: resp = await handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: return await handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: return await handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 339, in cached_function
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: response = await self.make_entry(
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 378, in make_entry
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: response = await handler(*args, **kwargs)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 237, in stream_content
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: return await self._match_and_stream(path, request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 544, in _match_and_stream
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: return await self._stream_content_artifact(
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 693, in _stream_content_artifact
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: response = await self._stream_remote_artifact(request, response, remote_artifact)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 943, in _stream_remote_artifact
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: download_result = await downloader.run()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 273, in run
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: return await download_wrapper()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: ret = await target(*args, **kwargs)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 258, in download_wrapper
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: return await self._run(extra_data=extra_data)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulp_rpm/app/downloaders.py", line 118, in _run
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: to_return = await self._handle_response(response)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 209, in _handle_response
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: await self.finalize()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 934, in finalize
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: await original_finalize()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/download/base.py", line 171, in finalize
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: self.validate_digests()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: File "/usr/lib/python3.9/site-packages/pulpcore/download/base.py", line 223, in validate_digests
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: raise DigestValidationError(actual_digest, expected_digest, url=self.url)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: pulpcore.exceptions.validation.DigestValidationError: A file located at the url http://localhost/pub/repo/bear-4.1-1.noarch.rpm failed validation due to checksum. Expected '7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3', Actual '1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e'
~~~

Expected results:

Client would receive an error instead of the bad file that does not match the metadata stored on the DB.

Additional info:

Steps described here are artificial (in order to facilitate reproducing it) but there are real use cases being hit by customers.

I personally observed this behavior on Satellite 6.11 and 6.13 (didn't tested other versions but I suppose they behave the same)