Major Description
0.16 release brought significant changes to the way we configure the brokers. The issue #2426 describes a bug during the upgrade which this PR attempts to fix. There are two way how users will usually run into an 0.16 upgrade.
- User has a version specified in his Kafka CR and set to 2.3.1. When the CO is upgraded to 0.16.0, it iwll run a regular reconciliation which updates everything to 0.16.0 but remains on 2.3.1. Only later the user changes the version field to Kafka 2.4.0. The CO will do the upgrade, the statefulset is already managed by 0.16.0 so there are no (known / reported) issues.
- The user does not have version field specified. In such case it is aways using the latest available Kafka version. With 0.15 this would be 2.3.1. When the CO is updated to 0.16, the Kafka upgrade to 2.4.0 will be happening immediately on the beginning of the first reconciliation. In this case the upgrade will need to deal with an old stateful set. This is a problem which was not handled in the code until this PR and is described in Strimzi#2426. This PR handles is by refreshing the whole Statefulset to the 0.16.0 level including all the other required resources (CM with broker config, new DNS names in broker certs etc.)
I also as another option considered to just detect these situations and enforce first an initial update using the lower version. However, this would not work when the ogirinal version was not 2.3.1 but for example Kafka 2.2.0 in which case we would basically do an upgrade during the rolling update. Another problem is that teh Zookeeper upgrade is independent process and is already done when we execute this. That is why the solution in this PR seemed better.
