Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-2208

Obsolete Tests Are Missing from OTE List and Metadata provided by OTE update

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

          1. Summary
            OTE's handling of test update and obsolete scenarios appears incomplete, especially in how it impacts Component Readiness and long-term test tracking.
          1. Context
            In `main.go`, we are using the `ext.IgnoreObsoleteTests()` function to mark tests as obsolete.  
            This allows us to:
      • Avoid validation errors during `OTE update`.
      • Ensure obsolete tests are removed from the metadata.
      • Proceed with code cleanup safely and unblock development.

      Example usage:

      // To delete tests you must mark them as obsolete.
      // These tests will be excluded from metadata validation during OTE update.
      // 1 - Run: OTE list to get the full test name
      // 2 - Add the test name here
      // 3 - Remove the test from your test file
      // 4 - Run OTE update
      ext.IgnoreObsoleteTests(
          "[sig-olmv1] OLMv1 should pass a trivial sanity check",
          // Add more removed test names below
      ) 

       

      This allows us to remove the test and have the metadata reflect that state.
      It’s useful because we can clean up code while keeping OTE update running smoothly.

      Problem

      However, several issues arise with this current approach:

      • Running OTE list does not show obsolete tests.
      • As a result, Component Readiness (or any consumer of this metadata) will have no visibility into tests that were recently removed.
      • This could lead to inconsistencies or require manual intervention from TRT to clean up or sync state with internal databases.

      Example output of OTE list after obsoleting a test:

      [
        {
          "name": "[sig-olmv1] OLMv1 should another pass a trivial sanity check",
          "labels": {},
          "resources": { "isolation": {} },
          "source": "openshift:payload:olmv1",
          "codeLocations": [
            "/Users/camilam/.../test/olmv1.go:12",
            "/Users/camilam/.../test/olmv1.go:13"
          ],
          "lifecycle": "blocking",
          "environmentSelector": {}
        }
      ] 

      The test [sig-olmv1] OLMv1 should pass a trivial sanity check is not present — not even as obsolete.

      Furthermore

      This also allows re-creating a test with the same name after it has been removed.
      However, it is unclear how this is handled:

      • Would it cause conflicts or confusion with test tracking?
      • Will Component Readiness treat it as a new test or a continuation of the old one?
      • Could this affect test ownership, lifecycle state, or metrics?

      Proposal

      Instead of removing obsolete tests from metadata and OTE list results, we suggest the following:

      • Obsolete tests should still be included in the metadata and listed by OTE list.
      • Each obsolete test should have a clear flag, e.g., "status": "obsolete" or similar.
      • Component Readiness and other systems can then recognize and handle these appropriately.

      This would:

      • Preserve test identity across time
      • Ensure auditability and consistency
      • Prevent confusion when names are reused
      • Allow downstream systems to track removals safely

      Completely removing obsolete tests from metadata does not seem appropriate, since we lose visibility and traceability — both of which are important for safe and predictable test lifecycle management.

       

              Unassigned Unassigned
              rh-ee-cmacedo Camila Macedo
              None
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: