The server, and Infinispan in general, should use the user marshaller to read application/octet-stream and application/unknown data.
ISPN-10433 added a hack in the server (for backwards compatibility, I assume): if jboss-marshalling is on the classpath, org.infinispan.server.core.LifecycleCallbacks.cacheManagerStarting() changes the marshaller for application/unknown (but not for application/octet-stream) to GenericJBossMarshaller.
This means that server/core, server/hotrod, and client/hotrod-client tests see application/unknown as protostream, but the server distribution in server/runtime always includes jboss-marshalling, so the server distribution sees application/unknown as GenericJBossMarshaller data.
It would be much better if the server always used the user marshaller for application/unknown, and when a user needs backwards compatibility between a new server and an old client, they should configure GenericJBossMarshaller in the server.
In addition, new clients should not use application/unknown unless the marshaller does not implement getMediaType(): they know the marshaller's media type, and they should send it to the server.