Proposed title of this feature request

Distributed tracing of platform components

What is the nature and description of the request?

The capability to configure tracing for platform components including (but not limited to) kube-apiserver, etcd, kubelet. Additionally any future components that are instrumented with OTel supported tracing should be configurable (openshift-apiserver, oauth). 

The trace destination must be configurable and could either be consumed by a provided OTel collector or by a $USER provided collector. Depending on the requirement to integrate with the OCP console itself. 

Collector should be configurable to enable a centralised observability model. 

https://kubernetes.io/docs/concepts/cluster-administration/system-traces/#kube-apiserver-traces 

Why does the customer need this? (List the business requirements)

SRE requires the following

• Move to a high cardinality/dimensionality data set that enables POST aggregation rather then pre which is the current form of metrics.
• Enables the ability to query at a wider scope (limited by K:Vs in traces, rather then labels in metrics or data in logs). This aids troubleshooting during an incident. 

• Enable high resolution analytics that assists in assessment of nominal performance

List any affected packages or components.

• kube-apiserver
• kubelet
• etcd

Notes

• Check syinergies with https://keptn.sh/stable/