-
Story
-
Resolution: Done
-
Major
-
None
User Story:
As a PSAP engineer I want to be able to load test the new caikit embedding endpoints. Usually we use llm-load-test for model serving load testing, however these new endpoints serve an entirely new purpose that doesn't conform to similar API's. Using llm-load-test is particularly helpful because we get the workload orchestration for free, so I'd like to be able to implement this testing in llm-load-test. I will try to follow the structure of llm-load-test as best as possible but will likely have to make large modifications since these new endpoints aren't doing text-generation.
Acceptance criteria:
An open fork (or potentially a PR) of llm-load-test that allows use to run an automated load test of the caikit embedding endpoints. This fork is not necessarily intended to be merged into llm-load-test.