Proposed title of this feature request

OpenTelemetry RHOAI Integration

What is the nature and description of the request?

OpenShift AI gives data scientists and developers a powerful AI/ML platform for building AI-enabled applications. Data scientists and developers can collaborate to move from experiment to production in a consistent environment quickly.

OpenShift AI is available as an add-on cloud service to Red Hat OpenShift Service on AWS or Red Hat OpenShift Dedicated or as a self-managed software product. It provides an AI platform with popular open source tooling. Familiar tools and libraries like Jupyter, TensorFlow, and PyTorch along with MLOps components for model serving and data science pipelines are integrated into a flexible UI.

As part of the OpenTelemetry Integrations framework delivered by the parent outcome (OBSDA-914), users shall be able to easily integrate Red Hat OpenShift AI with the OpenTelemetry ecosystem and vendors and be supported in the platform.

Why does the customer need this? (List the business requirements)

Users need to easily dpeloy and configure the observability pipelines of AI workloads by:

• Document the integration with configuration and examples
• Explore providing a dashboard in the OpenShift console related to relevant LLM info as defined in the OpenTelemetry standard.
  • Relevant info includes detecting hallucinations, text quality, length, word count... Read more here
  • Depending on the data, some of this data may be postponed to an integration with TrustyAI
• Enablement
  • Explore if it's possible to make RHOAI to install and/or configure an instrumentation CR
  • Explore if it's possible to make RHOAI to install and/or configure an OpenTelemetry collector to receive, process and export data inside and outside OpenShift
  • This may be a template like the one studied in this doc
• Introduce an observability section in https://ai-on-openshift.io/ 

List any affected packages or components.

• Red Hat build of OpenTelemetry
• RHOAI