Instrument Tekton reconciler code using OpenTelemetry and Jaeger

With distributed tracing, we can track the time taken by each action in the pipeline like reconciling logic, fetching resources, pulling images etc. This allows the developers to improve the reconciliation logic and also allow end users to monitor and optimize the pipelines.
 

Goals

• Implementation of opentelemetry tracing with Jaeger
• Instrumentation of pipelinerun and taskrun reconciliation logic
• Able to visualize pipeline and task reconciliation steps in jaeger

Requirements

• Trace all functions in the PipelineRun controller
• Trace all functions in the TaskRun controller
• Support Jaeger backend
• Propagate traces so that subsequent reconciles of the same resource belong to the same trace
• Propagate traces so that reconciles of a resource owned by a parent resource belong a parent span from the parent resource
• Reconcile of different resources must belong to separate traces

 
  *

• • (Optional) Use Cases{}

Pipeline and Task User:

• I would like to understand the duration of each task in my pipeline so that I can optimize the slow taks to improve the pipeline execution speed

Tekton Developer:

• I would like to understand the duration of each reconciliation step, so that I can optimize the code to improve reconciliation performance
• When the pipelines are failing due to a bug, I would like to understand which reconciliation logic caused the issue so that I can easily fix the problem

Out of scope

• Instrumentation of sidecars and initcontainers
• Support for more tracing backends
• Instrumentation of individual steps in each task
• Adding events to each span to indicate what is happening inside each method. This is an improvement task, and it can be done later once the basic setup for tracing is in place. The scope of this proposal is only to get the plumbing done by covering only the method boundaries.

Dependencies

< Link or at least explain any known dependencies. >

Background, and strategic fit

< What does the person writing code, testing, documenting need to know? >

Assumptions

< Are there assumptions being made regarding prerequisites and dependencies?>

< Are there assumptions about hardware, software or people resources?>

Related Architecture/Technical Documents

https://github.com/tektoncd/community/blob/main/teps/0124-distributed-tracing-for-tasks-and-pipelines.md