Uploaded image for project: 'Red Hat Enterprise Linux AI'
  1. Red Hat Enterprise Linux AI
  2. RHELAI-3616

Third-party model(s) support - for the end-to-end workflow and inference

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected
    • RHELAI-3557RHEL AI Third-Party Model Validation Deliverables for Summit '25

      Feature Overview (mandatory - Complete while in New status)
      An elevator pitch (value statement) that describes the Feature in a clear, concise way. ie: Executive Summary of the user goal or problem that is being solved, why does this matter to the user? The “What & Why”... 
       
      For 1.5, we are planning on supporting Llama 3.3 70b as the teacher model (student models are being discussed - not confirmed). For this phase, we expect the user flow to be: 
      1. At runtime, users specify the model ID, context length, system prompt. These pieces will need to be well-documented - the path and relevant context length specifically, as well as how to find/generate the model ID. 
      This will need to be specified once - when users run 'ilab data generate' or 'ilab model serve' the first time. 
      We expect the users to download these models from Quay or HuggingFace. This will need to be documented clearly. 

      The config file, today, has hard-coded values and paths for model path, context length etc. As we move towards multi-model support, this calls for the 'models' piece to be more dynamic. Upon model download - users get a model ID associated with the model. This model ID, when passed as a flag, will dynamically update the context length, model path, and system prompt. 

      Goals (mandatory - Complete while in New status)
      Provide high-level goal statement, providing user context and expected user outcome(s) for this Feature

      • Who benefits from this Feature, and how? 
      • What is the difference between today’s current state and a world with this Feature?

      <your text here>

      Requirements (mandatory -_ Complete while in Refinement status):
      A list of specific needs, capabilities, or objectives that a Feature must deliver to satisfy the Feature. Some requirements will be flagged as MVP. If an MVP gets shifted, the Feature shifts. If a non MVP requirement slips, it does not shift the feature.

      Requirement Notes isMVP?
           
           

       

      Done - Acceptance Criteria (mandatory - Complete while in Refinement status):
      Acceptance Criteria articulates and defines the value proposition - what is required to meet the goal and intent of this Feature. The Acceptance Criteria provides a detailed definition of scope and the expected outcomes - from a users point of view

      <your text here>

      Use Cases - i.e. User Experience & Workflow: (Initial completion while in Refinement status):
      Include use case diagrams, main success scenarios, alternative flow scenarios.
      <your text here>

      Out of Scope {}{}(Initial completion while in Refinement status):
      High-level list of items or persona’s that are out of scope. 

      1. For 1.5, any detailed work around config redesign or system redesign is out of scope
      2. We expect users to download the models themselves through Quay, for e2e and serve. 
      3. We'll rely on documentation to specify supported models for each of these workflows, and not look at feature gating right now.

      Documentation Considerations {}{}(Initial completion while in Refinement status):
      Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation..
      1. Listed above - all of this will need to be documented.

      2. Document clearly which ones are supported as teacher/student models and which ones are inference-only. 

      Questions to Answer {}{}(Initial completion while in Refinement status):
      Include a list of refinement / architectural questions that may need to be answered before coding can begin.
      1. Chat template changes and dynamic chat template detection and use? 

      2. Backward compatibility - we will have to ask users to regenerate the config file if they upgrade from 1.4 -> 1.5

      Background and Strategic Fit (Initial completion while in Refinement status):

      From an eng/runtime perspective, some of this (i.e. model path, context length, model family etc) is specified in the config today. So, we will have to make changes to that accommodate this new flow. 
       

      Customer Considerations {}{}(Initial completion while in Refinement status):
      Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.
      <your text here>

      Team Sign Off (Completion while in Planning status)

      • All required Epics (known at the time) are linked to the this Feature
      • All required Stories, Tasks (known at the time) for the most immediate Epics have been created and estimated
      • Add - Reviewers name, Team Name
      • Acceptance == Feature as “Ready” - well understood and scope is clear - Acceptance Criteria (scope) is elaborated, well defined, and understood
      • Note: Only set FixVersion/s: on a Feature if the delivery team agrees they have the capacity and have committed that capability for that milestone
      Reviewed By Team Name Accepted Notes
             
             
             
             

       

              jepandit@redhat.com Jehlum Vitasta Pandit
              jepandit@redhat.com Jehlum Vitasta Pandit
              Jaideep Rao, Kelly Brown
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: