Description of problem:

I have a number of docker repositories with download policy on_demand, which have accumulated content (blobs) for months. As I don't need the older versions of the images anymore, I ran reclaim-space on these repositories, but I'm unable to sync them again afterwards.

Version-Release number of selected component (if applicable):
Satellite 6.13.3 on RHEL 8.8

How reproducible:
Always. Reproduced in 2 different environments.

Steps to Reproduce:

1. Create a product:
[root@satellite ~]# hammer product create --name "RHOSP16 Containers"
Product created.
[root@satellite ~]#

2. Create a docker repository inside the product:
[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" --content-type docker --download-policy on_demand --upstream-username $username --upstream-password $password --url "https://registry.redhat.io" --docker-upstream-name rhosp-rhel8/openstack-aodh-base --name aodh-base
Repository created.
[root@satellite ~]#

3. Verify sync completes successfully:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
Added Docker tags: 89
Total steps: 1377/1377
--------------------------------
Associating Content: 1121/1121
Downloading Artifacts: 166/166
Downloading tag list: 1/1
Processing Tags: 89/89
Un-Associating Content: 0/0
[root@satellite ~]#

4. Reclaim space on the repository:
[root@satellite ~]# hammer repository reclaim-space --product "RHOSP16 Containers" --name aodh-base
Could not reclaim the repository:
undefined local variable or method `repositories' for #<Actions::Pulp3::Repository::ReclaimSpace:0x00007f0d552968f0>
Did you mean? repository_hrefs
[root@satellite ~]#

5. Since the previous step didn't work using hammer, I went to the web interface, selected the repository and clicked on "Reclaim Space" in "Select Action". This task completed successfully

6. Attempting to sync afterwards:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
No content added.
Total steps: 1/90
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading tag list: 1/1
Processing Tags: 0/89
Error: 'NoneType' object has no attribute 'file'
[root@satellite ~]#
[root@satellite ~]# hammer --output json repository info --product "RHOSP16 Containers" --name aodh-base | jq .Sync
{
"Status": "Warning",
"Last Sync Date": "1 minute"
}
[root@satellite ~]#

Actual results:
Sync fails

Expected results:
Sync works

Additional info:

Repository configuration:

[root@satellite ~]# hammer repository info --product "RHOSP16 Containers" --name aodh-base
Id: 106
Name: aodh-base
Label: aodh-base
Description:
Organization: LocalOrg
Red Hat Repository: no
Content Type: docker
Mirroring Policy: Content Only
Url: https://registry.redhat.io
Publish Via HTTP: yes
Published At: satellite.local.lab/localorg-rhosp16_containers-aodh-base
Relative Path: localorg-rhosp16_containers-aodh-base
Download Policy: on_demand
Upstream Repository Name: rhosp-rhel8/openstack-aodh-base
Container Image Tags Filter:
Container Repository Name: localorg-rhosp16_containers-aodh-base
HTTP Proxy:
HTTP Proxy Policy: global_default_http_proxy
Product:
Id: 303
Name: RHOSP16 Containers
GPG Key:

Sync:
Status: Warning
Last Sync Date: 1 minute
Created: 2023/10/11 11:10:00
Updated: 2023/10/11 11:11:57
Content Counts:
Container Image Manifest Lists: 72
Container Image Manifests: 144
Container Image Tags: 89

[root@satellite ~]#

Traceback in messages:

[root@satellite ~]# grep 18522 /var/log/messages
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 11c9fdaf-ce0f-4567-ade5-d0712d19029b
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=aodh-base-1063035 remote=aodh-base-106854
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:47 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.downloaders:INFO: Updating bearer token
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Task 11c9fdaf-ce0f-4567-ade5-d0712d19029b failed ('NoneType' object has no attribute 'file')
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: result = func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/synchronize.py", line 41, in synchronize
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: return dv.create()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: loop.run_until_complete(pipeline)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: return future.result()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: await asyncio.gather(*futures)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in _call_
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: await self.run()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 152, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: listed_manifest = await listed_manifest_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib64/python3.9/asyncio/tasks.py", line 611, in _wait_for_one
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: return f.result() # May raise f.exception().
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 448, in create_listed_manifest
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: saved_artifact, content_data = await sync_to_async(_get_content_data_blocking)()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 435, in _call_
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: ret = await asyncio.wait_for(future, timeout=None)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: return await fut
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: result = self.fn(*self.args, **self.kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 476, in thread_handler
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: return func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 444, in _get_content_data_blocking
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: content_data = json.load(saved_artifact.file)