Description of problem:

I have a number of docker repositories with download policy on_demand, which have accumulated content (blobs) for months. As I don't need the older versions of the images anymore, I ran reclaim-space on these repositories, but I'm unable to sync them again afterwards.


Version-Release number of selected component (if applicable):
Satellite 6.13.3 on RHEL 8.8

How reproducible:
Always. Reproduced in 2 different environments.

Steps to Reproduce:

1. Create a product:
[root@satellite ~]# hammer product create --name "RHOSP16 Containers"
Product created.
[root@satellite ~]# 

2. Create a docker repository inside the product:
[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" --content-type docker --download-policy on_demand --upstream-username $username --upstream-password $password --url "https://registry.redhat.io" --docker-upstream-name rhosp-rhel8/openstack-aodh-base --name aodh-base
Repository created.
[root@satellite ~]# 

3. Verify sync completes successfully:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
Added Docker tags: 89
Total steps: 1377/1377
--------------------------------
Associating Content: 1121/1121
Downloading Artifacts: 166/166
Downloading tag list: 1/1
Processing Tags: 89/89
Un-Associating Content: 0/0
[root@satellite ~]# 

4. Reclaim space on the repository:
[root@satellite ~]# hammer repository reclaim-space --product "RHOSP16 Containers" --name aodh-base 
Could not reclaim the repository:
  undefined local variable or method `repositories' for #<Actions::Pulp3::Repository::ReclaimSpace:0x00007f0d552968f0>
  Did you mean?  repository_hrefs
[root@satellite ~]# 

5. Since the previous step didn't work using hammer, I went to the web interface, selected the repository and clicked on "Reclaim Space" in "Select Action". This task completed successfully

6. Attempting to sync afterwards:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
No content added.
Total steps: 1/90
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading tag list: 1/1
Processing Tags: 0/89
Error: 'NoneType' object has no attribute 'file'
[root@satellite ~]# 
[root@satellite ~]# hammer --output json repository info --product "RHOSP16 Containers" --name aodh-base | jq .Sync
{
  "Status": "Warning",
  "Last Sync Date": "1 minute"
}
[root@satellite ~]# 


Actual results: 
Sync fails

Expected results:
Sync works


Additional info:

Repository configuration:

[root@satellite ~]# hammer repository info --product "RHOSP16 Containers" --name aodh-base
Id:                          106
Name:                        aodh-base
Label:                       aodh-base
Description:                 
Organization:                LocalOrg
Red Hat Repository:          no
Content Type:                docker
Mirroring Policy:            Content Only
Url:                         https://registry.redhat.io
Publish Via HTTP:            yes
Published At:                satellite.local.lab/localorg-rhosp16_containers-aodh-base
Relative Path:               localorg-rhosp16_containers-aodh-base
Download Policy:             on_demand
Upstream Repository Name:    rhosp-rhel8/openstack-aodh-base
Container Image Tags Filter: 
Container Repository Name:   localorg-rhosp16_containers-aodh-base
HTTP Proxy:                  
    HTTP Proxy Policy: global_default_http_proxy
Product:                     
    Id:   303
    Name: RHOSP16 Containers
GPG Key:                     

Sync:                        
    Status:         Warning
    Last Sync Date: 1 minute
Created:                     2023/10/11 11:10:00
Updated:                     2023/10/11 11:11:57
Content Counts:              
    Container Image Manifest Lists: 72
    Container Image Manifests:      144
    Container Image Tags:           89

[root@satellite ~]# 


Traceback in messages:

[root@satellite ~]# grep 18522 /var/log/messages
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 11c9fdaf-ce0f-4567-ade5-d0712d19029b
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=aodh-base-1063035 remote=aodh-base-106854
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:47 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.downloaders:INFO: Updating bearer token
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Task 11c9fdaf-ce0f-4567-ade5-d0712d19029b failed ('NoneType' object has no attribute 'file')
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    result = func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/synchronize.py", line 41, in synchronize
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return dv.create()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    loop.run_until_complete(pipeline)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return future.result()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    await asyncio.gather(*futures)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    await self.run()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 152, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    listed_manifest = await listed_manifest_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/tasks.py", line 611, in _wait_for_one
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return f.result()  # May raise f.exception().
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 448, in create_listed_manifest
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    saved_artifact, content_data = await sync_to_async(_get_content_data_blocking)()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 435, in __call__
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    ret = await asyncio.wait_for(future, timeout=None)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return await fut
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    result = self.fn(*self.args, **self.kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 476, in thread_handler
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 444, in _get_content_data_blocking
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    content_data = json.load(saved_artifact.file)

QE Tracker for https://issues.redhat.com/browse/SAT-20711
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2243256