Introduce delayed initialization strategy

Synopsis

The logmanager eagerly initializes as soon as logging is first used. In many cases this is undesirable because of circular dependency problems which do not allow the logging configuration to be fully available until other systems are booted for a variety of reasons:

• WildFly logging configuration is done in a subsystem which is not available at first boot
• Logging handlers may depend on other systems (which use logging) to initialize (like Elytron for SSL)
• Java 9+ may make it difficult to initialize logging very early

So we need a multi-stage initialization process.

Requirements

Requirement 1: Performance must not be impacted

One of the primary reasons for the existence of the log manager is performance. Therefore, performance impact must be as minimal as possible during the initialization process and effectively zero post-initialization.

Requirement 2: Process in correct order

Late initialization must allow all log messages to be processed in a correct order. Note that there is more than one possible correct order, however any log message B that was logged observably after log message A must appear in the log after A.

Requirement 3: No lost messages

When initialization is complete, all log messages which occurred before initialization must be logged and none lost.

Requirement 4: No mutations

Messages recorded during initialization cannot be mutated in any way compared to what would have been logged post-initialization; therefore, any log message produced before initialization is complete must have a full copy of its contextual information (caller and MDC/NDC). Note that this requirement is in natural tension with Requirement 1.

Requirement 5: Tracing

Capturing full context on every log record, including those at TRACE level, will certainly impose too severe a performance impact for normal usage. A reasonable optimization would be to choose a "safe" level like "INFO" during the initialization process, thereby requiring full capture of only a small number of log records. However this would in turn make debugging during this early stage quite difficult. So, it must be possible to inform the log manager that boot logging must accept messages of any valid level, including TRACE or ALL. The default should lean towards performance, but it should be easily overridable (say, via a system property).

Requirement 6: Opt-in

In most common cases, like unit testing, it is desirable to simply start logging from the properties configuration. So, delayed initialization should be opt-in as it is generally a container or advanced feature, and the application in question likely has to take action in order to complete logging initialization.

Implementation strategy

A possible implementation strategy would be to create a layer of indirection, centrally managed through the log manager instance, through which all logging is dispatched. This could be done by way of one of the org.jboss.logmanager.Logger#logRaw or org.jboss.logmanager.LoggerNode#publish methods.

State machine

The layer of indirection could have multiple stages:

• Uninitialized:
  • All log records are enqueued
  • The root logger level is initialized to the level configured by the initial system property
• Initializing:
  • All log records are enqueued
  • Loggers and handlers are configured by the container or user as they wish
  • At the end of this stage, an "initializationComplete" method on the log manager is called, causing a transition to...
• Starting:
  • All log records are enqueued as long as the queue is not "clear", at which point log records are dispatched normally
  • At the same time, the queue is played back in order and the log messages are dispatched according to their final configuration
  • When the queue is empty, it is atomically tagged as "clear", and the state is changed to...
• Running:
  • All log records are dispatched normally

Note that Requirement 3 can only be met if the configured initial log level is finer than the minimum level in the final configuration.

Safety checks

The initialization message queue should have a configurable (via system property again) limit. A default of, say, 10,000 messages would not be unreasonable and would allow for fairly fine-grained logging. If the cap of enqueued messages is reached, the error handler should report (one time) that the cap was reached and that some messages were lost, and it should suggest that either the level system prop be made coarser, or the limit should be increased, in order to prevent the situation from occurring.

No delayed initialization desired

A system property should be added to determine whether the log manager delays its initialization. Per requirements, it should default to false. If true, then default configuration should not be done at all; it is assumed in this case that the user will perform configuration tasks.