Uploaded image for project: 'Performance and Scale for AI Platforms'
  1. Performance and Scale for AI Platforms
  2. PSAP-1466

Implement Load Test for Caikit Embedding

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Major Major
    • RHOAI_2.12.0
    • None
    • RHOAI
    • 3
    • PSAP - General-8

      User Story:
      As a PSAP engineer I want to be able to load test the new caikit embedding endpoints. Usually we use llm-load-test for model serving load testing, however these new endpoints serve an entirely new purpose that doesn't conform to similar API's. Using llm-load-test is particularly helpful because we get the workload orchestration for free, so I'd like to be able to implement this testing in llm-load-test. I will try to follow the structure of llm-load-test as best as possible but will likely have to make large modifications since these new endpoints aren't doing text-generation.

      Acceptance criteria:
      An open fork (or potentially a PR) of llm-load-test that allows use to run an automated load test of the caikit embedding endpoints. This fork is not necessarily intended to be merged into llm-load-test.

              rh-ee-dripberg Drew Ripberger
              rh-ee-dripberg Drew Ripberger
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 3 minutes
                  3m
                  Remaining:
                  Remaining Estimate - 3 minutes
                  3m
                  Logged:
                  Time Spent - Not Specified
                  Not Specified