Uploaded image for project: 'Performance and Scale for AI Platforms'
  1. Performance and Scale for AI Platforms
  2. PSAP-1466

Implement Load Test for Caikit Embedding

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Major Major
    • RHOAI_2.12.0
    • None
    • RHOAI
    • PSAP - General-8

      User Story:
      As a PSAP engineer I want to be able to load test the new caikit embedding endpoints. Usually we use llm-load-test for model serving load testing, however these new endpoints serve an entirely new purpose that doesn't conform to similar API's. Using llm-load-test is particularly helpful because we get the workload orchestration for free, so I'd like to be able to implement this testing in llm-load-test. I will try to follow the structure of llm-load-test as best as possible but will likely have to make large modifications since these new endpoints aren't doing text-generation.

      Acceptance criteria:
      An open fork (or potentially a PR) of llm-load-test that allows use to run an automated load test of the caikit embedding endpoints. This fork is not necessarily intended to be merged into llm-load-test.

            rh-ee-dripberg Drew Ripberger
            rh-ee-dripberg Drew Ripberger
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 3 minutes
                3m
                Remaining:
                Remaining Estimate - 3 minutes
                3m
                Logged:
                Time Spent - Not Specified
                Not Specified