+++ This bug was initially created as a clone of Bug #2241934 +++

Description of problem:

I have a custom product on this satellite:
https://repo.skype.com/rpm/stable

This has worked well for years but it appears that Skype recently switched to ztd compression for its repodata.
Even though my Satellite has the RHEL8.8 zst packages installed, this makes the repo unsyncable and an error is seen in the sync task log:

Errors:
Cannot open /var/lib/pulp/tmp/6323@sat6.lasthome.solace.krynn/tmpdfzm7dwc/tmp6vz8wmzq-44e0a07b586dafa046deee582d4d49601f6b407703f53e0353546444cec4a78b-primary.xml.zst: Cannot detect compression type

Version-Release number of selected component (if applicable):
Satellite 6.15.1 + RHEL 8.8

How reproducible:
100%, the product and its CV are no longer updating since Skype switched to zst comnpression.

Steps to Reproduce:
1. Create a skype product (yum) with a yum repo pointing to: https://repo.skype.com/rpm/stable
2. Try to sync it.
3.

Actual results:
Errors:
Cannot open /var/lib/pulp/tmp/6323@sat6.lasthome.solace.krynn/tmpdfzm7dwc/tmp6vz8wmzq-44e0a07b586dafa046deee582d4d49601f6b407703f53e0353546444cec4a78b-primary.xml.zst: Cannot detect compression type

this is even though my Sat has the rpms installed:
[root@sat6 ~]# rpm -qa pulpcore* zst satellite
satellite-6.12.5.1-1.el8sat.noarch
pulpcore-selinux-1.3.2-1.el8pc.x86_64
zstd-1.4.4-1.el8.x86_64
libzstd-1.4.4-1.el8.i686
libzstd-devel-1.4.4-1.el8.x86_64
libzstd-1.4.4-1.el8.x86_64

— Additional comment from on 2023-10-03T14:07:28Z

I meant 6.12.5.1, sorry, not 6.15.1!

— Additional comment from on 2023-10-03T16:51:49Z

We have in-progress upstream PR. Once merged it can be backported.

— Additional comment from on 2023-10-03T17:07:16Z

@bbuckingham@redhat.com @ehelms This may end up requiring async releases as this is likely to be just the first of many repos which switch to using zstd compression. It's not a regression on our side, but rather a feature which we don't currently support, which is showing up in the wild for the first time due to a recent change of defaults for createrepo_c. createrepo_c recently added support (within the past few months) and then released with it as a new default (very recently, 2 months ago).

Thus this is technically adding a feature, but without which certain repos may stop being able to be synced should they decide to switch.

DNF has supported it since I believe RHEL 8.2 so it's not a new concept, it's just new that it's actually being used. It went from there being no way to easily produce such metadata to it being the default for some tools in a short period.

— Additional comment from on 2023-10-04T03:02:19Z

So this is not currently supported, but is very easy to add support for. Just a couple of lines and a dependency update. See: https://github.com/pulp/pulp_rpm/pull/3255/files

I don't know how aggressively we want to backport it but I just want to make clear that whether repos update or not to zstd metadata, is not within our control, and therefore repos could in theory just start switching and "breaking" out of the blue. I don't think that will happen quickly, certainly not with the important ones, but still my suggestion is that we backport back to at least 6.12 to avoid getting caught out by it.

That would require an async update, but I kind of expect we will want one anyway for some of the import/export issues which we are working on currently.

— Additional comment from on 2023-10-04T19:48:46Z

Hi Daniel, thank you for chiming in. I would love to see this resolved on 6.12.z in an async or a z-stream release.

— Additional comment from on 2023-10-08T23:42:06Z

Fix also requires a new version of createrepo_c, 1.0.1

— Additional comment from on 2023-10-09T16:06:01Z

The Pulp upstream bug status is at closed. Updating the external tracker on this bug.

— Additional comment from on 2023-11-13T12:59:55Z

Hello Daniel,

My customer, ZITiS - Zentrale Stelle für Informationstechnik im Sicherheitsbereich, would like to know which Red Hat Satellite release will support this feature: 6.12.z, 6.13.z or 6.14.z.
They will decide which Satellite release to upgrade to based on this information.

Thank you,

Mohamed

— Additional comment from on 2023-11-13T14:46:41Z

That's a question for the triage folks, not for me

It's backported to the upstream builds for the 6.12, 6.13 and 6.14 lines and I've added request flags already, it just needs to actually be accepted.

— Additional comment from on 2023-11-13T14:48:44Z

Granted, I'm sure we'd prefer them to upgrade to 6.14 than 6.12 regardless of whether we backport this to 6.12 eventually.

— Additional comment from on 2023-11-13T14:53:08Z

Thanks a lot, for your reply, Daniel.

— Additional comment from on 2023-12-11T09:39:55Z

Verified in 6.15.0 snap 1.0 (python3.11-pulp-rpm-3.23.0-2.el8pc.noarch):
1. Ensured https://repo.skype.com/rpm/stable still uses the zst compression.
2. Successfully synced it on Satellite - all packages were synced.

— Additional comment from on 2024-01-03T14:00:42Z

Hello,

My customer, ZITiS - Zentrale Stelle für Informationstechnik im Sicherheitsbereich, reached out again asking if we have any plans to backport this feature to 6.12.z and 6.13.z.
Would you kindly advise?

Thank you,

Mohamed

— Additional comment from on 2024-01-03T14:48:44Z

Odilon - I know we don't plan any future z-streams for 6.12. What about 6.13?

— Additional comment from on 2024-01-04T14:47:10Z

// RH internal

dear Satellite Product Management

some context;

• for my current customer (CU), as we have to move off 6.11 by end of month anyway, I can live with this RFE not making it into 6.12.z and 6.13.z I just wanted a Q asked a couple weeks back in the access ticket to be answered. That's what prompted Comment #13
• for some of my previous CUs, if it is not too much hassle for Sat eng to add zst support, please do backport (comment #9 gives me thhe impression this really is not haasle at all on the Eng side).
  Some of these previous CUs have been slow to upgrade to the next .y version in the past but upgrading to the latest .z has been better
  and as correctly stated in comment #4, Sat CUs will see a breakage if a repo (outside or RH and CU control) switches to the modern compression type.

So, if possible, do support zstd on 6.13 and if not too much hassle to 6.12 as well. But do not consider the RFE a must have for ZITiS, it is a nice to have for previous CUs I delivered to.

Kind regards,

Patrick (one of the Hatters delivering at ZITiS at the moment)

— Additional comment from on 2024-01-04T15:49:51Z

(In reply to Patrick C. F. Ernzer from comment #15)
> // RH internal
>
> Patrick (one of the Hatters delivering at ZITiS at the moment)

Hi Patrick, the main problem to backport to 6.12.z is that we don't have bandwidth to verify the changes, to enable zst we need to update the createrepo_c package and this involves us also providing the system python(3.6) version of createrepo_c on RHEL8, we plan to ship this on 6.13.z, we delayed until 6.14.1 GA because we don't want to create drift between the new stable version and the old stable, this can cause upgrade problems.

(In reply to Jeremy Lenz from comment #14)
> Odilon - I know we don't plan any future z-streams for 6.12. What about 6.13?

Hi Jeremy, we have plans to add this one 6.13.z ASYNC, we had to wait until 6.14.1 release to avoid regressions.

Regards,
Odilon

QE Tracker for https://issues.redhat.com/browse/SAT-22344
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2257300