-
Bug
-
Resolution: Done
-
Critical
-
None
-
None
-
5
-
False
-
Sat perf&scale Sprint 16
-
None
-
None
-
None
-
None
Background
Pablo pinged me:
I'm officially on PTO, but I wanted to let you know that maybe we (as in the Satellite product) have a problem Take a look at https://jenkins-csb-perf-master.dno.corp.redhat.com/job/ContPerfStreamEL9/, the last two builds have taken more than 6h more than expected (I cannot verify it ATM because the VPN is acting funny). I've checked 6.17 results from last run and seem consistent We didn't have Stream results for almost ten days because a bug in Satellite that blocked our testing, so if this behaviour persists, it's something that has been introduced in Satellite since Stream 113 (112 was fine and is the code that corresponds with the latest stable resutls) https://reportportal-perf.apps.dno.ocp-hub.prod.psi.redhat.com/ui/#satcpt/launches/all/30364/215324/215330/log?logParams=history%3D214804%26page.page%3D1 (example with 180 concurrent registrations) and this one showing a 20% increase overall in average: https://reportportal-perf.apps.dno.ocp-hub.prod.psi.redhat.com/ui/#satcpt/launches/all/30364/215324/215347/log?logParams=history%3D214821%26page.page%3D1
So I compared duration of each phase of these two jobs:
### https://jenkins-csb-perf-master.dno.corp.redhat.com/job/ContPerfStreamEL9/145/consoleText Start at: 2025-07-14T10:22:55+00:00 [+0:0:0] ===== Setup ===== [+0:46:44] ===== Checking environment ===== [+0:5:17] ===== Create base LCE(s), CCV(s) and AK(s) ===== [+0:15:2] ===== Prepare for Red Hat content ===== [+0:4:54] ===== Sync OS from CDN ===== [+0:34:42] ===== Create, publish and promote OS CVs / CCVs to LCE(s)s ===== [+0:14:26] ===== Push OS content to capsules ===== [+0:32:10] ===== Publish and promote big CV ===== [+0:3:26] ===== Publish and promote filtered CV ===== [+0:3:32] ===== Get Satellite Client content ===== [+0:0:54] ===== Create, publish and promote Satellite Client CVs / CCVs to LCE(s)s ===== [+0:6:20] ===== Push Satellite Client content to capsules ===== [+0:0:59] ===== Get RHOSP content ===== [+0:8:22] ===== Create, publish and promote RHOSP CVs / CCVs to LCE(s)s ===== [+0:6:53] ===== Push RHOSP content to capsules ===== [+0:1:0] ===== Get Flatpak content ===== [+0:1:2] ===== Create, publish and promote Flatpak CVs / CCVs to LCE(s)s ===== [+0:7:0] ===== Push Flatpak content to capsules ===== [+0:1:0] ===== Sync yum repo ===== [+0:14:16] ===== Push yum content to capsules ===== [+0:0:55] ===== Sync iso ===== [+0:15:44] ===== Push iso content to capsules ===== [+0:0:56] ===== Sync docker repo ===== [+0:12:56] ===== Push docker content to capsules ===== [+0:0:55] ===== Sync ansible collections ===== [+0:5:59] ===== Push ansible collections content to capsules ===== [+0:0:57] ===== Prepare for registrations ===== [+0:4:45] ===== Incremental registrations ===== [+5:36:26] ===== Misc simple tests ===== [+0:16:3] ===== BackupTest ===== [+0:36:15] ===== Remote execution (ReX) ===== [+3:38:39] ===== Generate satellite-maintain report ===== [+0:0:22] ===== Delete all content hosts ===== [+0:34:53] ===== Delete base LCE(s), CCV(s) and AK(s) ===== [+0:9:9] ===== Sosreport =====
### https://jenkins-csb-perf-master.dno.corp.redhat.com/job/ContPerfStreamEL9/153/consoleText Start at: 2025-07-25T04:54:03+00:00 [+0:0:0] ===== Setup ===== [+0:47:17] ===== Checking environment ===== [+0:5:17] ===== Create base LCE(s), CCV(s) and AK(s) ===== [+0:14:44] ===== Prepare for Red Hat content ===== [+0:4:57] ===== Sync OS from CDN ===== [+0:35:17] ===== Create, publish and promote OS CVs / CCVs to LCE(s)s ===== [+0:14:10] ===== Push OS content to capsules ===== [+0:31:57] ===== Publish and promote big CV ===== [+0:3:26] ===== Publish and promote filtered CV ===== [+0:3:26] ===== Get Satellite Client content ===== [+0:0:54] ===== Create, publish and promote Satellite Client CVs / CCVs to LCE(s)s ===== [+0:6:22] ===== Push Satellite Client content to capsules ===== [+0:0:59] ===== Get RHOSP content ===== [+0:8:5] ===== Create, publish and promote RHOSP CVs / CCVs to LCE(s)s ===== [+0:6:43] ===== Push RHOSP content to capsules ===== [+0:0:59] ===== Get Flatpak content ===== [+0:1:11] ===== Create, publish and promote Flatpak CVs / CCVs to LCE(s)s ===== [+0:7:0] ===== Push Flatpak content to capsules ===== [+0:1:0] ===== Sync yum repo ===== [+0:7:59] ===== Push yum content to capsules ===== [+0:0:54] ===== Sync iso ===== [+0:6:6] ===== Push iso content to capsules ===== [+0:0:55] ===== Sync docker repo ===== [+0:5:41] ===== Push docker content to capsules ===== [+0:0:56] ===== Sync ansible collections ===== [+0:6:2] ===== Push ansible collections content to capsules ===== [+0:0:58] ===== Prepare for registrations ===== [+0:4:45] ===== Incremental registrations ===== [+7:4:49] ===== Misc simple tests ===== [+0:16:31] ===== BackupTest ===== [+0:18:44] ===== Remote execution (ReX) ===== [+9:31:40] ===== Generate satellite-maintain report ===== [+0:0:22] ===== Delete all content hosts ===== [+0:35:18] ===== Delete base LCE(s), CCV(s) and AK(s) ===== [+0:9:9] ===== Sosreport =====
We can see "Incremental registrations" section duration changed from 05:36:26 to 07:04:49 and "Remote execution (ReX)" changed from 03:38:39 to 09:31:40.
Also failure rate changed drastically.
Acceptance criteria
- Investigate difference in duration and failure rate of most important tests in "Incremental registrations" and "Remote execution (ReX)" section.
- If this is a product bug, report it and make sure to stress it is clear it is a regression.
- If it is a issue in our automation, fix it.
- relates to
-
SAT-36317 [Regression] - Seen difference in duration and failure rate of most important tests in "Incremental registrations" and "Remote execution (ReX)"
-
- Closed
-