Uploaded image for project: 'Multiple Architecture Enablement'
  1. Multiple Architecture Enablement
  2. MULTIARCH-3682

Research why the s390x upgrade jobs are slower in CI

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Obsolete
    • Icon: Normal Normal
    • None
    • None
    • IBM Z CI
    • None
    • False
    • None
    • False
    • NEW
    • s390x
    • NEW

      A bug was opened a while ago asking for more information on why s390x jobs had gotten slower.

      Original bug's details (bug linked below as well):

      Upgrades for s390x appear to have gotten more than 30 minutes slower in the last few days, specifically for this job periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-nightly-4.13-ocp-ovn-remote-libvirt-s390x

 timestamp  |   percentile_50    |   percentile_99    |   percentile_100  
------------+--------------------+--------------------+--------------------
 2023-04-03 |  85.17597379908334 |  85.17597379908334 |  85.17597379908334 
 2023-04-06 |  94.53841424391666 |  94.53841424391666 |  94.53841424391666 
 2023-04-12 |     101.0090252604 |     101.0090252604 |     101.0090252604 
 2023-04-13 |  88.40349969108334 |  88.40349969108334 |  88.40349969108334 
 2023-04-14 |      92.8558838846 |      92.8558838846 |      92.8558838846 
 2023-04-15 |  87.67548067665001 |  87.67548067665001 |  87.67548067665001 
 2023-04-16 |  95.20680148158334 |  95.20680148158334 |  95.20680148158334 
 2023-04-17 |  89.00616086978333 |  89.00616086978333 |  89.00616086978333 
 2023-04-19 |     111.8592981407 |     111.8592981407 |     111.8592981407 
 2023-04-20 |  93.67446999196666 |  93.67446999196666 |  93.67446999196666 
 2023-04-22 |    114.17386392945 |    114.17386392945 |    114.17386392945 
 2023-04-27 |  95.17302858490001 |  95.17302858490001 |  95.17302858490001 
 2023-04-29 |     91.74251922565 |     91.74251922565 |     91.74251922565 
 2023-04-30 | 124.67553157201668 | 124.67553157201668 | 124.67553157201668 
 2023-05-01 | 119.67508063240001 | 119.67508063240001 | 119.67508063240001 
 2023-05-02 | 117.50727737876667 | 117.50727737876667 | 117.50727737876667 
 2023-05-03 | 118.50686850626667 | 118.50686850626667 | 118.50686850626667

       

      After some initial investigation and manual upgrade testing, the upgrades were completed successfully.  Since the difference in information is pretty severe (manual upgrade taking around 70 minutes, while CI averages being greater than that), more in depth investigation is required.

      Completion Criteria:

      • Information regarding the increased upgrade time within CI detailed via comments on this task or in an external doc (google doc, etc)
      • Applicable bugs or tasks filed in Jira to address any specific issues identified

            Unassigned Unassigned
            tvardema Trevor Vardeman
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: