+++ This bug was initially created as a clone of Bug #2170485 +++

Description of problem:

• Customer importing Red Hat Repository on Disconnected Red Hat Satellite taking huge time around 5 hours.

Version-Release number of selected component (if applicable):

• Red Hat Satellite Version 6.11.4

How reproducible:

• Try to import Rhel 7 or Rhel 8 AppsSream or Rel 8 BaseOs repository sync will complete but it will take huge time (Approximate 5-6 hours)

Steps to Reproduce:

1. Customer having 36 GB Ram and 8 CPU.

2. Import Huge Red Hat Repository on Disconnected Satellite and observe the time taken to complete.

Actual results:

• Importing huge repository takes 5 hours

Expected results:

• Customer want to improve this and need time similar to Repository sync.

Additional info:

• NA

— Additional comment from on 2023-02-16T14:14:15Z

Adding some extra context:

Complete import of a big repository is expected to take a long time. However, an incremental (which is a lot smaller, having only a few hundred MBs) also takes a long time to complete.

Example from an internal test.

Complete import of AppStream 8 (85GB tarball): 6h 35min
Started at: 2023-02-06 19:10:44 UTC
Ended at: 2023-02-07 01:45:02 UTC

Incremental import of same repository (190 MB tarball): 3h 15min
Started at: 2023-02-07 21:51:36 UTC
Ended at: 2023-02-08 01:05:24 UTC

Here are some numbers (and words) from the customer:

=====
I run a full import of 84G appstream repo and it takes 5 hours 13 mins:
[root@satellitesync121 1.0]# time hammer content-import repository -organization="SATSYNC121" --path=/var/lib/pulp/imports/Export-Red_Hat_Enterprise_Linux_8_for_x86_64__AppStream_RPMs_8-54/1.0/2023-01-24T19-01-55-05-00
[..................................................................................................................] [100%]
real 313m19.000s
user 1m34.717s
sys 0m12.542s

Then, I run an incremental import of 77M appstream repo and it takes 2 hours 27 min:
[root@satellitesync121 2023-01-25T09-27-11-05-00]# time hammer content-import repository -organization="SATSYNC121" --path=/var/lib/pulp/imports/Export-Red_Hat_Enterprise_Linux_8_for_x86_64__AppStream_RPMs_8-54/2.0/2023-01-25T09-27-11-05-00
[..................................................................................................................] [100%]
real 147m40.602s
user 0m47.357s
sys 0m6.578s
=====

Customer's numbers match with mine: an incremental import takes around half the time of a complete import.

The goal of this BZ here is to try getting some improvement on the incremental import times.

— Additional comment from on 2023-02-16T15:23:36Z

There was a perf-related fix in core/3.21 for import - see https://github.com/pulp/pulpcore/issues/3075. This hasn't been backported, but def could be. How difficult would it be to get a test-scenario in-house, where we could apply this patch "by hand" and see if it helps w/ the specific scenario the customer is involved in?

— Additional comment from on 2023-02-16T16:18:56Z

Not difficult at all. I have it already in place, where I ran my tests to confirm what the customer was saying.

If you can get me a patch I'll gladly apply there and run the imports again. It will take a while (or not, if the patch really works well )

— Additional comment from on 2023-02-16T16:41:35Z

(In reply to Joniel Pasqualetto from comment #3)
> Not difficult at all. I have it already in place, where I ran my tests to
> confirm what the customer was saying.
>
> If you can get me a patch I'll gladly apply there and run the imports again.
> It will take a while (or not, if the patch really works well )

OK - will need to squeeze out some time and see if the change backports cleanly, will ping you when I know more.

— Additional comment from on 2023-02-20T15:12:28Z

Created attachment 1945283
Patch backporting pulp-#3075 to core/3.16

Attaching a patch to backport the import-improvement to core/3.16. Let's test and see if it helps any.

— Additional comment from on 2023-02-20T19:17:53Z

I took the liberty to apply the patch in one of my labs and run a quick test.

Both times are on the same server. Second import (with the patch) was done after disabling the imported repository and cleaning up orphans.

Original import time: 114.17 minutes
Started at: 2023-02-09 21:08:32 UTC
Ended at: 2023-02-09 23:02:42 UTC

Import time after patch: 91.53 minutes
Started at: 2023-02-20 16:43:32 UTC
Ended at: 2023-02-20 18:15:04 UTC

I will run a similar test on a Satellite 6.12 too (got the patch for core/3.18 upstream). This one will take longer (slower hardware).

— Additional comment from on 2023-02-20T20:11:16Z

@jpasqual@redhat.com 20% improvement is something. Let us know how your 6.12 test goes. I've merged the upstream backport into both core/3.16 and core/3.18 branches.

— Additional comment from on 2023-02-21T21:47:11Z

The environment where my 6.12 was hosted is suffering from a performance issue. I had to spin a new one somewhere else.

I also decided to run a new set of tests on the 6.11 and include a complete and an incremental imports.

For each Satellite version, the content being imported (with and without the patch) is the same. A complete cleanup (including deleting orphans) was executed between each complete import.

Content imported on 6.11 is not exactly the same as the content imported on 6.12 (but pretty similar).

Here the results:

1. 1. 1. Satellite 6.11 - ORIGINAL CODE ###

Complete import: 117.3 minutes
Started at: 2023-02-21 15:08:22 UTC
Ended at: 2023-02-21 17:05:40 UTC

Incremental import: 58.15 minutes
Started at: 2023-02-21 20:05:35 UTC
Ended at: 2023-02-21 21:03:44 UTC

1. 1. 1. Satellite 6.11 - PATCHED CODE ###

Complete import: 90.53 minutes
Started at: 2023-02-20 16:43:32 UTC
Ended at: 2023-02-20 18:15:04 UTC

Incremental: 32.05 minutes
Started at: 2023-02-21 13:35:22 UTC
Ended at: 2023-02-21 14:07:25 UTC

1. 1. 1. Satellite 6.12 - ORIGINAL CODE ###

Complete import: 108.03 minutes
Started at: 2023-02-21 15:02:53 UTC
Ended at: 2023-02-21 16:50:55 UTC

Incremental import: 56.68 minutes
Started at: 2023-02-21 17:51:03 UTC
Ended at: 2023-02-21 18:47:44 UTC

1. 1. 1. Satellite 6.12 - PATCHED CODE ###

Complete import: 83.2 minutes
Started at: 2023-02-21 19:20:14 UTC
Ended at: 2023-02-21 20:43:26 UTC

Incremental import: 30.35 minutes
Started at: 2023-02-21 21:00:40 UTC
Ended at: 2023-02-21 21:31:01 UTC

— Additional comment from on 2023-02-21T21:55:48Z

@jpasqual@redhat.com these results look good, great test runs thank you!

This patch is already backported into the 3.16 and 3.18 branches for pulpcore (the fix was originally released in core/3.21). This should make this available for hotfixing if desired. I will push the "release z-stream" button for each upstream branch so we can release in an erratum when/if/as that becomes desirable.

— Additional comment from on 2023-02-23T04:16:44Z

I released z-streams upstream and I'm moving this to POST