Story

Create 500+ single-run Q&A dataset against RHDH 1.9.
categorize the Q&A's use same topic list: https://github.com/redhat-ai-dev/lightspeed-evaluation-dataset/blob/main/categories_rhdh.yaml

Background

will continue use ragas library to generate the datasets with RHDH 1.9 release doc.

Dependencies and Blockers

RHDH 1.9 docs need to be released. Developer Lightspeed RAG should have the 1.9 docs image & start consuming it.

Acceptance Criteria

created 500 single-run Q&A dataset with questions, context & expected answers, in the format of evaluation_data yaml that the lightspeed evaluation tool is expected

publish the dataset in git repo: https://github.com/redhat-ai-dev/lightspeed-evaluation-dataset

Note: the dataset will be AI-generated, human validation will be out of scope