Problem statement

customer has a performance issue, app is running slower, or a benchmark is slower compared to say, the same benchmark on VMware. There are a lot of dimensions and configurations that can influence the storage performance of a VM. Trying them all is not possible, and frustrating for a customer with an active performance escalation.

Currently they use esxtop. It shows a latency view of the whole storage stack of a VM in a single place, and you could determine which layer was the main contributor to latency.

see attach graphic:

https://www.codyhosterman.com/wp-content/uploads/2018/03/stack1.png

• CMDS/s    total IOps
• DAVG/cmd    average device RT (ms) per command
• KAVG/cmd    the time the command spends in VMkernel.
• GAVG/cmd    RT as perceived by the guest
• formula: DAVG + KAVG = GAVG

Analysis

1. Do we have similar performance metrics in our guest / VM / CSI storage stack?
2. Is there an easy way to consume them?

Note: This analysis is predicated in "Little's Law", and requires 3 metrics at each storage layer.

1. completion rate (IOps)
2. service time (latency in ms)
3. queue depth

source slack thread 

User Experience Considerations

Starting with a command line tool that can be run. Based on the results, creating a UI can be considered, where the tool is run in a container and results are provided in graphical form. However, for the time being, no UXD support is needed.