Goal:

Validate third-party models in InstructLab flows and Inference flows on vLLM for Summit by doing inference performance benchmarking and accuracy evaluations for third-party models to give customers flexibility, confidence and predictability bringing third-party models to Instruct Lab and vLLM within RHEL AI.

• Implement Llama 3.3 70B Teacher Support in InstructLab for RHEL AI 1.5
  • inference covered here with i lab serve test
• Enable Granite 3.1 8B base starter v2 (base) Student Support in InstructLab for RHEL AI 1.5
  • inference covered here with i lab serve test
• Inference testing w/ ilab download, ilab model list, and ilab serve for:
  • the models listed here: https://docs.google.com/spreadsheets/d/1NGPhJV0pk7jYuAFOHk7aWPomX7Svb_-Xa-OVUVtpNbM/edit?gid=1505755754#gid=1505755754 - validated model list
• Models packaged and validated for use on Quay for InstructLab flows 

Acceptance Criteria:

• Instruct Lab flows enable users to swap the teacher out (pulled in from Quay) to Llama 3.3 70B, student should remain as Granite 3.1 8B
• InstructLab flows enable users to swap the student out (pulled in from Quay) to Llama 3.1 8B, teacher should remain as Mixtral 7x8B
• Give users confidence deploying inference w/: 
  • https://docs.google.com/spreadsheets/d/1NGPhJV0pk7jYuAFOHk7aWPomX7Svb_-Xa-OVUVtpNbM/edit?gid=1505755754#gid=1505755754 - validated model list
• End-to-end functional tests are completed in RHEL AI 1.5 
• Note: the inference performance benchmarks and general evaluation (OpenLLMLeaderboard v1/v2 ONLY) will be done by the PSAP team. 
  • We don't need to show dk-bench results/significant accuracy improvements for Summit Timeline.
  • The story is around flexibility and optionality to BYOM (not that you get better performance using these third-party models...yet)