Description of problem:

There are certain upstream providers who don't need filelists in their repository metadata and hence they publish the whole repository content and it's metadata without the filelists.xml file. 

But pulp-3 in Red Hat Satellite\Capsule 6.10+ , is not able to sync any repository whose metadata is missing other.xml or filelists.xml .

How reproducible:

100%

Is this issue a regression from an earlier version:

No

Steps to Reproduce:

1. Create a custom repo ( with SSL validation disabled ) using upstream URL https://artifacts.elastic.co/packages/8.x/yum 

2. Sync the repo in satellite

 

Actual behavior:

  "error":

{     "traceback": "  File \"/usr/lib/python3.11/site-packages/pulpcore/tasking/tasks.py\", line 66, in _execute_task\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 568, in synchronize\n    repo_version = dv.create() or repo.latest_version()\n                   ^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 161, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.11/asyncio/base_events.py\", line 654, in run_until_complete\n    return future.result()\n           ^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/api.py\", line 220, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/api.py\", line 41, in __call__\n    await self.run()\n  File \"/usr/lib/python3.11/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 873, in run\n    await self.parse_repository_metadata(repomd, repomd_files)\n  File \"/usr/lib/python3.11/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 910, in parse_repository_metadata\n    raise FileNotFoundError(\n",     "description": "XML file(s): filelists not found"   }

,

 

Expected behavior:

Non-RedHat repositories should be allowed to sync in Pulp, if they are missing the filelists.xml file in their upstream  repo metadata.

 

A fix exists in the upstream pulp_rpm: https://github.com/pulp/pulp_rpm/commit/6abc7e783a3e2496387eb0bdade8473f4561da39 

So we need to pull in both of these fixes for pulp_rpm and createrepo_c into satellite and capsule.

Allow incremental parsing without filelists or other xml by dralley - Pull Request #441 - rpm-software-management/createrepo_c: https://github.com/rpm-software-management/createrepo_c/pull/441

Enable syncing without filelists present by dralley - Pull Request #3780 - pulp/pulp_rpm: https://github.com/pulp/pulp_rpm/pull/3780

 

 

Business Impact / Additional info:

Recent discussions:

Missing repodata/filelists.xml in yum repo causes repo sync to fail - Elastic Stack / Elasticsearch - Discuss the Elastic Stack: https://discuss.elastic.co/t/missing-repodata-filelists-xml-in-yum-repo-causes-repo-sync-to-fail/374168

Hazelcast RPM repository fails to sync with Redhat Satellite due to missing filelist in repomod.xml - Issue #182 - hazelcast/hazelcast-packaging: https://github.com/hazelcast/hazelcast-packaging/issues/182

External Private Yum Repository on JFrog.io not synchronizing in Foreman/Katello - Support - TheForeman: https://community.theforeman.org/t/external-private-yum-repository-on-jfrog-io-not-synchronizing-in-foreman-katello/24938