-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
6.13.7.3, 6.12.3, 6.14.4.3, 6.16.0, 6.15.5
-
2
-
False
-
-
False
-
0
-
Phoenix
-
-
-
Sprint 140, Sprint 141
-
Moderate
-
To Do
-
Yes
Description of problem:
Trying to fix corrupted metadata of a repository on Capsule, an attempt to run Full Capsule Sync does not help. The reason is Pulp sync task returns "Skipping Sync (no change from previous sync)" - and this is caused by the fact katello always call `RepositoriesRpmApi.sync` endpoint with `"optimize":true`. That is caused by:
https://github.com/Katello/katello/blob/master/app/lib/actions/pulp3/capsule_content/sync.rb#L16-L17
That needs to be:
{noblock}sync_options[:optimize] = false if input[:options][:skip_metadata_check]
output[:pulp_tasks] = repo.backend_service(smart_proxy).with_mirror_adapter.sync(sync_options){noblock}
How reproducible:
100%
Is this issue a regression from an earlier version:
Yes, a regression from 6.9
Steps to Reproduce:
1. Have one CV promoted to LE that is associated to a Capsule and synced there. Ideally have just one pulp repository on the Capsule, to simplify the next step.
2. Manually corrupt some metadata on filesystem on the Capsule - e.g. via `file $(find /var/lib/pulp/media/artifact/ -type f)`, identify `primary.xml` file and remove it.
3. Have a Content Host registered via the Capsule and enabled the newly-broken repo. Run there e.g. `dnf update` to confirm metadata are missing. Check `/var/log/messages` on Capsule for the details.
4. Run Full Capsule Sync
5. Once it completes, try `dnf update` again.
Actual behavior:
3. `dnf update` fails with something like:
zoo_repo 304 B/s | 55 B 00:00 Errors during downloading metadata for repository 'RedHat_zoo_product_zoo_repo': - Status code: 500 for https://pmoravec-caps615.example.com/pulp/content/RedHat/RECYCLE/cv_zoo/custom/zoo_product/zoo_repo/repodata/repomd.xml (IP: 1.2.3.4) Error: Failed to download metadata for repo 'RedHat_zoo_product_zoo_repo': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
`/var/log/messages` on Capsule shows:
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: [2024-10-04 08:58:55 +0000] [2043] [ERROR] Error handling request
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: Traceback (most recent call last):
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: resp = await request_handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 543, in _handle
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: resp = await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib64/python3.11/site-packages/aiohttp/web_middlewares.py", line 114, in impl
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: return await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib/python3.11/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: return await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 339, in cached_function
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: response = await self.make_entry(
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 378, in make_entry
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: response = await handler(*args, **kwargs)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 249, in stream_content
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: return await self._match_and_stream(path, request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 610, in _match_and_stream
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: return await self._serve_content_artifact(ca, headers, request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 918, in _serve_content_artifact
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: raise Exception(_("Expected path '{}' is not found").format(path))
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: Exception: Expected path '/var/lib/pulp/media/artifact/3b/bd4af9a5cac6195a2b138e9754df0c3d8acf7cdf34ed736bc313342409aa45' is not found
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content[2043]: [04/Oct/2024:08:58:55 +0000] "GET /pulp/content/RedHat/RECYCLE/cv_zoo/custom/zoo_product/zoo_repo/repodata/repomd.xml HTTP/1.1" 500 245 "-" "libdnf (Red Hat Enterprise Linux 8.0; generic; Linux.x86_64)"
So far, so good.
BUT the same persists also after Full Capsule Sync - that will have "Skipping Sync (no change from previous sync)" message in the pulp task details (see e.g. in Actions::Pulp3::CapsuleContent::Sync dynflow step of the task). Step 5 will fail the same way..
Expected behavior:
4. Full Caps Sync should not skip syncing the repo in Actions::Pulp3::CapsuleContent::Sync step.
5. `dnf update` should not fail after the Full Caps Sync
Business Impact / Additional info:
This prolonged resolution of a support case where we had to manually remove or fix individual repos instead of simply clicking to Full Caps Sync (which was tried, of course)
