Story (Required)

As a developer/user trying to trigger a Tekton pipeline from a repository with many PipelineRun files I want the pipeline to trigger quickly, similar to repositories with fewer files, so that my development workflow is not significantly delayed.

This story aims to address the performance degradation observed when triggering Tekton pipelines from GitHub repositories containing a large number of `PipelineRun` files in the `.tekton` directory. Currently, 70 `PipelineRun` files can increase trigger time from ~12 seconds to over a minute, significantly impacting user experience and development efficiency. The goal is to optimize this process to reduce the triggering time.

Background (Required)

Customer is experiencing slow pipeline triggering times for their GitHub repositories with many `PipelineRun` files. Specifically, a repository with 70 `PipelineRun` files in the `.tekton` directory takes over a minute to trigger, while a repository with only one `PipelineRun` file takes approximately 12 seconds. This issue is observed on Tekton version 1.19.3. It is currently unclear if this is expected behavior, but there's a clear need for optimization using the 70 `PipelineRun` file baseline.

Out of scope

• Investigating other performance bottlenecks not directly related to the number of `PipelineRun` files.
• Optimizing for repository structures other than GitHub with a `.tekton` directory.
• Deep-dive into the underlying Git clone/checkout performance (unless directly identified as being impacted by `PipelineRun` file processing).

  Approach (Required)

The general approach will involve identifying and optimizing the specific steps in the pipeline triggering process that are directly proportional to the number of `PipelineRun` files. This may include:
1. Profiling: Analyze the current triggering mechanism to pinpoint where the time is being spent when many `PipelineRun` files are present (e.g., resource discovery, parsing, validation).
2. Optimization Strategies: Explore and implement solutions such as:

• Implementing caching mechanisms for `PipelineRun` definitions.
• Optimizing file scanning/loading logic for greater efficiency.
• Considering lazy loading or on-demand parsing of `PipelineRun` files.
• Investigating internal Tekton changes to reduce overhead associated with processing numerous `PipelineRun` resources.

3. Baseline: Use a repository with 70 `PipelineRun` files as the primary baseline for performance measurement and improvement goals.
Further technical details, including potential json schemas or class definitions, will be elaborated during the design and implementation phase.

Dependencies

• Access to a testing environment mirroring the customer's setup (e.g., Tekton 1.19.3, GitHub repository with a large number of `PipelineRun` files).
• Potential collaboration or insights from the core Tekton project team regarding resource processing mechanisms.

  Acceptance Criteria (Mandatory)

• Given a GitHub repository with 70 `PipelineRun` files in the `.tekton` directory, when a pipeline is triggered, then the triggering time should be significantly reduced compared to the current ~60+ seconds.
• Given a GitHub repository with 70 `PipelineRun` files in the `.tekton` directory, when a pipeline is triggered, then the triggering time should meet an agreed-upon performance target (e.g., < 20 seconds, or a minimum percentage reduction).
• The optimization should not negatively impact the triggering time for repositories with a small number of `PipelineRun` files (e.g., ~12 seconds for 1 file).
• The solution must maintain the correct functionality and execution of all `PipelineRun` files without introducing regressions.
• Performance metrics and measurements must be captured to quantitatively demonstrate the improvement.

INVEST Checklist

Dependencies identified

Blockers noted and expected delivery timelines set

Design is implementable

Acceptance criteria agreed upon

Story estimated

Legend

Unknown

Verified

Unsatisfied

Done Checklist

• Code is completed, reviewed, documented and checked in
• Unit and integration test automation have been delivered and running cleanly in continuous integration/staging/canary environment
• Continuous Delivery pipeline(s) is able to proceed with new code included
• Customer facing documentation, API docs etc. are produced/updated, reviewed and published
• Acceptance criteria are met