Today the MCO has a strict alphanumeric ordering for MachineConfigs. This means if multiple machineconfigs define the same file, when the merging happens, only the definition in the highest alphanumeric order takes effect, and all other definitions are discarded.

This works pretty well generally but runs into trouble for generated machineconfigs, for example, those generated by kubelet or containerruntime config controllers. Since those generate machineconfigs with the pool naming (e.g. 99-generated-$POOL-kubeletconfig) this becomes problematic when you have a custom pool definition as well as a worker pool definition, because worker generally overrides custom pool due to most pool names starting before w.

This has caught a few customers, recently resulting in the incident https://docs.google.com/document/d/1P7lasC1ta9Zp_6h0rUYRG2uErk9351qCfz6klIEVeUk/edit#heading=h.ef3ucx201pqh 

There are a few options we can do, but one of the more straightforward options would be to always allow custom pool configs to supercede worker configs of the same file. Since if you were defining something in a custom pool, it likely is intended as special configuration for that pool.

Alternatively, we can rework the kubelet/containerruntime config controllers to handle this (and other parts) better. There is an old stub card https://issues.redhat.com/browse/MCO-84 that somewhat references this issue, but I think that would be more effort