Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-36238

Investigate reg and ReX time regression in SatStreamEL9_daily

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • None
    • Performance
    • 5
    • False
    • Sat perf&scale Sprint 16
    • None
    • None
    • None
    • None

      Background

      Pablo pinged me:

      I'm officially on PTO, but I wanted to let you know that maybe we (as in the Satellite product) have a problem
      Take a look at https://jenkins-csb-perf-master.dno.corp.redhat.com/job/ContPerfStreamEL9/, the last two builds have taken more than 6h more than expected (I cannot verify it ATM because the VPN is acting funny). I've checked 6.17 results from last run and seem consistent
      We didn't have Stream results for almost ten days because a bug in Satellite that blocked our testing, so if this behaviour persists, it's something that has been introduced in Satellite since Stream 113 (112 was fine and is the code that corresponds with the latest stable resutls)
      https://reportportal-perf.apps.dno.ocp-hub.prod.psi.redhat.com/ui/#satcpt/launches/all/30364/215324/215330/log?logParams=history%3D214804%26page.page%3D1 (example with 180 concurrent registrations)
      and this one showing a 20% increase overall in average: https://reportportal-perf.apps.dno.ocp-hub.prod.psi.redhat.com/ui/#satcpt/launches/all/30364/215324/215347/log?logParams=history%3D214821%26page.page%3D1
      

      So I compared duration of each phase of these two jobs:

      ### https://jenkins-csb-perf-master.dno.corp.redhat.com/job/ContPerfStreamEL9/145/consoleText
      Start at: 2025-07-14T10:22:55+00:00
      [+0:0:0] ===== Setup =====
      [+0:46:44] ===== Checking environment =====
      [+0:5:17] ===== Create base LCE(s), CCV(s) and AK(s) =====
      [+0:15:2] ===== Prepare for Red Hat content =====
      [+0:4:54] ===== Sync OS from CDN =====
      [+0:34:42] ===== Create, publish and promote OS CVs / CCVs to LCE(s)s =====
      [+0:14:26] ===== Push OS content to capsules =====
      [+0:32:10] ===== Publish and promote big CV =====
      [+0:3:26] ===== Publish and promote filtered CV =====
      [+0:3:32] ===== Get Satellite Client content =====
      [+0:0:54] ===== Create, publish and promote Satellite Client CVs / CCVs to LCE(s)s =====
      [+0:6:20] ===== Push Satellite Client content to capsules =====
      [+0:0:59] ===== Get RHOSP content =====
      [+0:8:22] ===== Create, publish and promote RHOSP CVs / CCVs to LCE(s)s =====
      [+0:6:53] ===== Push RHOSP content to capsules =====
      [+0:1:0] ===== Get Flatpak content =====
      [+0:1:2] ===== Create, publish and promote Flatpak CVs / CCVs to LCE(s)s =====
      [+0:7:0] ===== Push Flatpak content to capsules =====
      [+0:1:0] ===== Sync yum repo =====
      [+0:14:16] ===== Push yum content to capsules =====
      [+0:0:55] ===== Sync iso =====
      [+0:15:44] ===== Push iso content to capsules =====
      [+0:0:56] ===== Sync docker repo =====
      [+0:12:56] ===== Push docker content to capsules =====
      [+0:0:55] ===== Sync ansible collections =====
      [+0:5:59] ===== Push ansible collections content to capsules =====
      [+0:0:57] ===== Prepare for registrations =====
      [+0:4:45] ===== Incremental registrations =====
      [+5:36:26] ===== Misc simple tests =====
      [+0:16:3] ===== BackupTest =====
      [+0:36:15] ===== Remote execution (ReX) =====
      [+3:38:39] ===== Generate satellite-maintain report =====
      [+0:0:22] ===== Delete all content hosts =====
      [+0:34:53] ===== Delete base LCE(s), CCV(s) and AK(s) =====
      [+0:9:9] ===== Sosreport =====
      
      ### https://jenkins-csb-perf-master.dno.corp.redhat.com/job/ContPerfStreamEL9/153/consoleText
      Start at: 2025-07-25T04:54:03+00:00
      [+0:0:0] ===== Setup =====
      [+0:47:17] ===== Checking environment =====
      [+0:5:17] ===== Create base LCE(s), CCV(s) and AK(s) =====
      [+0:14:44] ===== Prepare for Red Hat content =====
      [+0:4:57] ===== Sync OS from CDN =====
      [+0:35:17] ===== Create, publish and promote OS CVs / CCVs to LCE(s)s =====
      [+0:14:10] ===== Push OS content to capsules =====
      [+0:31:57] ===== Publish and promote big CV =====
      [+0:3:26] ===== Publish and promote filtered CV =====
      [+0:3:26] ===== Get Satellite Client content =====
      [+0:0:54] ===== Create, publish and promote Satellite Client CVs / CCVs to LCE(s)s =====
      [+0:6:22] ===== Push Satellite Client content to capsules =====
      [+0:0:59] ===== Get RHOSP content =====
      [+0:8:5] ===== Create, publish and promote RHOSP CVs / CCVs to LCE(s)s =====
      [+0:6:43] ===== Push RHOSP content to capsules =====
      [+0:0:59] ===== Get Flatpak content =====
      [+0:1:11] ===== Create, publish and promote Flatpak CVs / CCVs to LCE(s)s =====
      [+0:7:0] ===== Push Flatpak content to capsules =====
      [+0:1:0] ===== Sync yum repo =====
      [+0:7:59] ===== Push yum content to capsules =====
      [+0:0:54] ===== Sync iso =====
      [+0:6:6] ===== Push iso content to capsules =====
      [+0:0:55] ===== Sync docker repo =====
      [+0:5:41] ===== Push docker content to capsules =====
      [+0:0:56] ===== Sync ansible collections =====
      [+0:6:2] ===== Push ansible collections content to capsules =====
      [+0:0:58] ===== Prepare for registrations =====
      [+0:4:45] ===== Incremental registrations =====
      [+7:4:49] ===== Misc simple tests =====
      [+0:16:31] ===== BackupTest =====
      [+0:18:44] ===== Remote execution (ReX) =====
      [+9:31:40] ===== Generate satellite-maintain report =====
      [+0:0:22] ===== Delete all content hosts =====
      [+0:35:18] ===== Delete base LCE(s), CCV(s) and AK(s) =====
      [+0:9:9] ===== Sosreport =====
      

      We can see "Incremental registrations" section duration changed from 05:36:26 to 07:04:49 and "Remote execution (ReX)" changed from 03:38:39 to 09:31:40.

      Also failure rate changed drastically.

      Acceptance criteria

      • Investigate difference in duration and failure rate of most important tests in "Incremental registrations" and "Remote execution (ReX)" section.
      • If this is a product bug, report it and make sure to stress it is clear it is a regression.
      • If it is a issue in our automation, fix it.

              rhn-support-ikaur Imaanpreet Kaur
              jhutar-1 Jan Hutaƙ
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: