Goal:

• Create a repository containing the first 1000 question-and-answer pairs for evaluating the performance of our RAG system. These pairs will be synthetically generated and subsequently validated by OpenStack experts to ensure the data we use for evaluations is valid.
• Our data points should be stored in a human-readable format within the repository (YAML). To ensure clear organization, the data must be logically divided based on its source (for example, based on the documentation section)
• Each data point should have a metadata assigned that indicate whether a given data point was synthetically generated or not, whether it was human validated (more to be defined)

Acceptance Criteria:

• A dedicated repository exists containing an initial dataset of 1000 synthetically generated, expert-validated question-and-answer pairs.

Open questions:

• What metadata do we want to store for individual data points?
• What is the best approach for synthetic dataset generation, the question / answer pairs?
• What is going to be a process for OpenStack teams to contribute to this repository?
  • We should think about this, but fully opening the repository for human written question / answer pairs should be a follow-up epic, probably.