Current implementation doesn't allow definition of independent properties - instead they will always have form of "mail.$ServerName.someprop". Thus it's not possible to setup properties such as "mail.transport.protocol"[1] inside custom server.

Also customer will unlikely design custom transport protocol from scratch. More often it will be just a wrapper around existing protocol such as smtp with some additional functionality (this is also case of [2]). However as a workaround one can define custom server together with server for underlying protocol within one mail session.

[1] which will cause session.getTransport(); to return null as default protocol is not set.
[2] https://c.na7.visual.force.com/apex/Case_View?id=500A000000A4N8b