What

BrokerState metric should be exposed to Prometheus and added to a dashboard so that the support team can understand the state of the broker. 

Why

The broker state reveals the current internal state of the broker.  This important to understand the state of the service.   This is critical information for the SRE when trying to diagnose problems with the service.

• The state the broker is in when it first starts up NOT_RUNNING((byte) 0)
• The state the broker is in when it is catching up with cluster metadata. STARTING((byte) 1)
• The broker has caught up with cluster metadata, but has not yet been unfenced by the controller. RECOVERY((byte) 2)
• The state the broker is in when it has registered at least once, and is  accepting client requests.   RUNNING((byte) 3)
• The state the broker is in when it is attempting to perform a controlled  shutdown.   PENDING_CONTROLLED_SHUTDOWN((byte) 6)
•  The state the broker is in when it is shutting down.  SHUTTING_DOWN((byte) 7),
• The broker is in an unknown state. UNKNOWN((byte) 127)

How

• Expose the Kafka JMX mbean to Prometheus: https://github.com/bf2fc6cc711aee1a0c2a/kas-fleetshard/blob/main/operator/src/main/resources/kafka-metrics.yaml

Done

• Metric expose recorded in prometheus