Uploaded image for project: 'OpenShift Pipelines'
  1. OpenShift Pipelines
  2. SRVKP-7835

Implement Standardized Task Output Results for CheckRun results

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      Executive Summary

      Currently, Pipelines as Code (PaC) reports the status of a pipeline task to Git providers based on the final Tekton TaskRun status (`Succeeded`/`Failed`). However, some advanced CI workflows, adopted by downstream projects, use a model where tasks are designed to always succeed technically to allow pipelines to complete. These tasks report their true logical outcome (e.g., `ERROR`, `WARNING`) in a structured result. This creates a discrepancy where a task is marked as "successful" in a pull request while it actually found a critical issue.

      This feature proposes to generalize on an already adopted convention by having PaC inspect task results for a standardized output (e.g., `TEST_OUTPUT`) and use its content to set a more accurate status on the Git provider's check.

      Goals

      This feature benefits end-users and developers on any platform using this CI model by providing consistent and trustworthy status reporting. The current behavior is confusing when a user sees a green check in their merge request but a failure or warning in their platform's UI. By aligning the PaC status with the logical test result, we will:

      • Eliminate confusion and provide a single source of truth for a task's outcome directly in the MR/PR.
      • Improve the user experience by ensuring the UI, policy engines, and Git provider checks all reflect the same result (`SUCCESS`, `WARNING`, or `ERROR`).
      • Increase trust in the CI system by making statuses transparent and intuitive.

      Requirements

      Requirement Notes
      PaC inspects for a standardized result PaC should check if a completed TaskRun contains a result with a standard name (e.g., `TEST_OUTPUT`, based on existing conventions).
      Parse the standardized result JSON If found, PaC must parse the JSON content to access a `result` field.
      Map `result` to Git provider status The value of the `result` field should be used to determine the Git provider's check conclusion.
      Define status mapping
      • `result` of `ERROR` or `FAILURE` -> Maps to a `failure` conclusion.
      • `result` of `WARNING` -> Maps to a `neutral` conclusion.
      • `result` of `SUCCESS` -> Maps to a `success` conclusion.
      • `result` of `SKIPPED` -> Maps to a `skipped` conclusion.
      Prioritize structured result over TaskRun status If the standardized result is present, its outcome MUST override the generic TaskRun status (`Succeeded`) for reporting purposes.
      Maintain existing behavior if result is absent If a TaskRun does not have the standardized result, PaC should continue to use the TaskRun's overall status as it does today.

      Use Cases

      • Who: A developer using a CI platform that employs the "let pipelines proceed" model.
      • What: A `deprecated-base-image-check` task runs. The task itself succeeds (as designed by the pipeline), but its structured `TEST_OUTPUT` result is `"result":"ERROR", ...`.
      • Why/Problem: PaC currently reports "Success" to the Git provider because the TaskRun succeeded. This is misleading. With this feature, PaC would parse the structured result, see the `"result":"ERROR"`, and correctly report a "Failure" for that check in the merge request, matching the user's expectation.

      Out of Scope

      • This feature does not propose changing the behavior of Tekton itself.
      • This feature does not require all projects to adopt this single convention, but rather provides a generic mechanism in PaC that any project can use by conforming to the standard.

      Dependencies

      • This feature relies on a well-defined standard for structured results. The convention already adopted by some downstream projects (using `TEST_OUTPUT`) serves as a proven model.

      Background and Strategic Fit

      For CI/CD models like "let pipelines proceed" to be effective, result reporting must be accurate. The user support issue highlights a critical flaw in the current experience. Some downstream platforms have already solved this for their own ecosystems by defining a convention for structured results (`TEST_OUTPUT`).

      This feature is about generalizing that proven pattern. Instead of these platforms maintaining custom forks or requesting one-off logic in PaC, this change would build an abstracted, generic capability into PaC. This allows PaC to support these advanced workflows for any project without becoming tightly coupled to a specific one, making for a more robust and extensible solution.

      Assumptions

      • A clear standard for the result name and structure (e.g., based on the existing `TEST_OUTPUT` convention) can be agreed upon for this generic feature.
      • The PaC team has the necessary context and technical ability to implement logic that inspects Tekton results.

      Customer Considerations

      • This is a significant improvement for users of advanced CI/CD pipelines, as it makes feedback much clearer. It is a non-breaking, opt-in feature for all other users.

      Documentation Considerations

      • Updates to existing content: The Pipelines as Code documentation should be updated to explain this new behavior: how it inspects TaskRun results and how users can format their own task results to control the status of their checks.
      • Release Note: This should be a prominent item in the release notes, explaining the new capability for more granular status reporting.

      What does success look like?

      • When a task produces a structured result with `"result":"ERROR"`, the corresponding check in the GitHub/GitLab PR/MR is marked as failed (red X).
      • When a task produces a structured result with `"result":"WARNING"`, the check is marked as neutral.
      • When a task produces a structured result with `"result":"SUCCESS"`, the check is marked as successful (green check).
      • The original user's issue is resolved, and other projects can now adopt this pattern without further changes to PaC.

      QE Contact

      • QE will need to create test scenarios with tasks that produce the standardized structured results for each state and verify that PaC reports the correct corresponding status to the Git provider.

      Impact

      • This feature will directly impact the logic within Pipelines as Code for reporting statuses. It makes PaC more adaptable to different CI methodologies.

      Related Architecture/Technical Documents

      Done Checklist

      • [ ] Acceptance criteria are met.
      • [ ] Non-functional properties of the Feature have been validated (such as performance, resource, UX, security, or privacy aspects).
      • [ ] User Journey automation is delivered.
      • [ ] Support and SRE teams are provided with enough skills to support the feature in a production environment.

              rh-ee-vbobade Vibhav Bobade
              cboudjna@redhat.com Chmouel Boudjnah
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: