fmperf and fmwork are currently being used within IBM research to gather tons of performance data of different LLMs on different hardware / runtimes. It sounds like currently there is a lot of similarity between fmperf and llm-load-test and potential room to collaborate on an open source tool.

The first step is trying these out and seeing if we could use them within our current testing workflow