[1817057] ContainerDisk is sometimes OOMKilled on some systems

Description of problem:

On some systems we see that the containerDisk container gets OOMKilled quite regularly. The reason seems to be connected to dynamic memory management of golang and possibly things like the used kernel version. The golang memory spikes are harmless in general and not big, but they are big enough to sometimes hit the memory limit of 40 M on the containerDisk container.

We had once a case 6 months ago on Azure on kubevirt 0.20, where people reported that issue. There it was solved by bumping the limit to 40MB.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

VMs get on some nodes frequently restarted.

Expected results:

VMs should not be restarted.

Additional info:

It is possible to work around that by using a DataVolume and specify there the containerDisk as the import source. The possible disadvantages are more storage and bandwidth consumption on the distributed storage for ephemeral data which the VMs normally don't have to keep after restarts. The storage usage can be reduced by putting the resulting PVC as an ephemeral volume on the VM, so that one PVC can be used for multiple VMs, but it still puts more pressure on the distributed storage than the containerDisk.

We are proposing https://github.com/kubevirt/kubevirt/pull/2844 to fix this in kubevirt. We basically rewrite the containerDisk binary in C to have more guarantees regarding to the memory consumption. We could also increase the memory limit, but this would have significant impact on the ram usage.