Epic Goal

Integrate the existing Lightspeed Core evaluation tool with the RHDH Lightspeed plugin and establish a functional testing environment.

Why is this important?

• …

Scenarios

1. Investigate & POC: Research the lightspeed-evaluation tool and establish a proof-of-concept (POC) integration with the RHDH Lightspeed plugin.

1. Configure Environment: Set up the necessary configuration files (e.g., system.yaml) to define the evaluation environment, including the "Judge LLM" and API access.

1. Validate Connectivity: Perform an initial test run with a minimal dummy dataset to ensure the evaluation tool can successfully communicate with the plugin and get responses.

1. Define Test Harness: Document the end-to-end process for a developer to manually trigger an evaluation run from their local environment

Acceptance Criteria (Mandatory)

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• Acceptance criteria are met
• Non-functional properties of the Feature have been validated (such as performance, resource, UX, security or privacy aspects)
• User Journey automation is delivered
• Support and SRE teams are provided with enough skills to support the feature in production environment