When a key with value of size 1 Mb is stored into the grid, very large amounts of garbage is generated - on the order of 30 Mb.
This garbage eventually becomes unreferenced, but it is still a surprisingly large amount of garbage and adversely affects the size of the heap and promotes the need for garbage collection. It may be that some of this memory is being used unnecessarily, and even reducing the amount of memory used would probably improve performance.
Steps to reproduce:
1. Start a single node EDG cluster, instrumented for jProfiler
2. Remotely attach the jProfiler GUI to the node, with CPU and memory allocation turned off
3. Once the GUI is attached, start memory allocation recording and CPU recording
4. Use HotRod or memcached client to put a single key witha 1 Mb value to the node
5. Turn off recording
The Recorded Objects memory views show that roughly 30 Mb of data have been allocated (not solely the first time through either). The same results can be seen by using jmap -heap to view the overall heap stats.