The same improvements done for PLANNER-2404 are also desirable in case of toMap() and toSortedMap(). That said, they may prove impractical.
If/when we're doing this, we should also look into PLANNER-2405 but for sorted maps instead.