Uploaded image for project: 'OpenShift Pipelines'
  1. OpenShift Pipelines
  2. SRVKP-9988

PipelineRun is reconciled thousands of times causing performance issues

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      Description

      Upstream Issue: https://github.com/tektoncd/pipeline/issues/8495

      PipelineRuns are being reconciled thousands of times repeatedly while in progress, even when there are no changes. This causes excessive load on the controller and impacts performance.

      Expected Behavior

      Reconciliation should only happen when:

      • The PipelineRun changes
      • Child TaskRuns change
      • Other relevant resources change

      Actual Behavior

      The controller reconciles the same PipelineRun thousands of times continuously while it's in progress. Eventually the PipelineRun completes successfully, but the excessive reconciliation causes:

      • High CPU usage on the controller
      • Increased latency for other operations
      • Potential scalability issues

      Impact

      • Performance degradation on clusters with many PipelineRuns
      • Controller resource exhaustion
      • Affects scalability and production stability
      • May impact SLAs for pipeline execution times

      Evidence

      Controller logs show repeated reconciliation of the same PipelineRun without changes. See attached logs in the upstream issue.

      Environment

      • Tekton Pipelines: v0.66.0
      • Kubernetes: v1.30.6

      Notes

      Investigation needed to identify root cause and reproduce reliably.

              vdemeest Vincent Demeester
              vdemeest Vincent Demeester
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: