Look into caching role mapping results for the duration of an overall operation. This would be an impl detail of the RoleMapper. With our standard mappers, the mapping results should not be changing in the middle of an operation, so I believe they should be cacheable.
Likely place to cache them is as an attachment in a context object that will be passed in to the Authorizer. It would have to be passed in to the RoleMapper as well.