So, I tried to figure out how we could do the dynamic reloading of certificates when they change. The main challenges I think we have are: * When you update the certificates dynamically, you cannot use any startup scripts to prepare them (e.g. to create PKCS12 files, bundle multiple files into single file, do any validation checks, etc.)

• It is hard to rely on the certificates being mounted from a Secret as volumes because:

• You need to restart a pod to mount a new Secret or possibly to mount new items from an existing Secret
• While Kubernetes can update the mounted files when the Secret is updated, it is not fully deterministic when it happens because it is done by Kubernetes through some periodic checks. So, for the operator, it is hard to know when to trigger the reload of the certificate because it does not know if it was updated or not.

The easiest way to work around these limitations seems to be to use config providers to load the certificates directly from a Secret: * We will use the PEM files inlined directly in the configuration (e.g. ssl.keystore.key, ssl.keystore.certificate.chain etc.). But instead of inlining them into the config, we will set these options to a config provider that will load the data from the Secret

• That way, we work around the indeterministic updates of mounted volumes
• And we can prepare the data in the config provider (e.g. join multiple certificates into a bundle etc.)

• The use of config providers instead of inlining the certificates directly or loading them from files also might allow us in the future to load the certificates from other sources - e.g. proper secret stores such as Vault etc.

I was able to prototype this and the reloading works. But the main issue is that the Kafka describe configs API seems to always return: * null for all the sensitive fields such as passwords, SASL configs, certificates

• For non-sensitive fields, it return the values already rendered by the config provider and not the config provider configuration itself.

This makes the diffing of the configuration impossible because I have nothing to compare it against (And even if I would inline the certificates directly instead of using the config provider, we would still get the null value as those are sensitive fields). And because I cannot diff it, we end up updating the fields every reconciliation (~ every 2 minutes). It seems to work, but we are essentially reconfiguring the SSL listeners again and again for a change that happens only once in a while. I guess that is not optimal … !https://a.slack-edge.com/production-standard-emoji-assets/14.0/apple-medium/1f615@2x.png!So, I wonder if there is a way to bypass the two limitations and: * Get the value even for sensitive fields through the Admin API

• Get the original configuration value (i.e. the config provider configuration such as ${strimzicerts:custom-cert:user2.key}{}) instead of the rendered value provided by the config provider

Strictly speaking, we should not need the raw values for the sensitive fields such as seeing the actual password or certificate if we get original config-provider-based value (which at least in our case is not sensitive). Is there some workaround for getting these values? Would it be something what could be a possible improvement for a future Kafka release? 
 

https://redhat-internal.slack.com/archives/C0681K5E9M1/p1727270494672739