Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-251

Sippy: Exclude aggregated jobs from other variants

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • None
    • None
    • None

      Jobs that make up an aggregated run are already tracked in Sippy. We are
      injecting unwanted hysteresis that will cause changes to pass rates to
      lag behind. The effect is probably small, but I think it's worth being
      correct.

      Let's take a simple example:

      The follow test runs 10x and passes 7.

      [sig-pizza] Pizza should be delivered within 30 minutes

      It has a pass rate of 70%, in line with historical results.

      If it is aggregated, we have an 11th entry in Sippy of a pass. Sippy
      actually thinks the test passed 8 times out of 11 – a pass rate of
      72.7%.

              stbenjam Stephen Benjamin
              stbenjam Stephen Benjamin
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: