I'm proposing a refactoring of the huge ResteasyProviderFactory(impl) class. With the distinction of public spi and core, we can afford to split the class into smaller components without too much concerns of users having access to them. Moreover, the ResteasyProviderFactory really covers client and server aspects together, while it's quite common to need an instance for client side usage only that results in memory/cpu waste for building and populating multiple server-side-related collections. Splitting the ResteasyProviderFactory isolating client and server only aspects allows avoiding that inefficiency in some scenarios.
Besides that, whenever needing a new ResteasyProviderFactory on client side (to be wrapped into a LocalResteasyProviderFactory), instead of creating a new instance from scratch, we can cache and reuse the default ResteasyProviderFactory for each ThreadContextClassLoader (as the initial configuration is effectively function of the classloader, which controls the providers discovery); this should reduce memory allocation.
Speaking of memory allocation reduction, we can create few singletons for the HeaderDelegate that are added to almost all ResteasyProviderFactory instances.