+++ This bug was initially created as a clone of Bug #2052689 +++

Description of problem:

The overhead memory calculations that Openshift uses is incorrect.

The VMI's Pod is getting OOMed causing the VM guest to terminate.

The sum of the RSS memory within the VMI's pod exceeds the Pod's memory limit. This means that the sum of all the processes in this pod (virt-launcher, qemu, libvirt, etc...) somehow crosses the memory overhead we calculate being needed for the Pod which hosts the guest VM.

Version-Release number of selected component (if applicable):

4.x Current customer case takes place in 4.8

How reproducible:

Based on the information in this PR, I can see that it appears possible for the memory limit to be exceeded. Here's the evidence I've gathered.

When the VM has a request of 22 Gi set on the VM's spec, the corresponding pod request is 22.23 Gi. That extra .23 GI on the pod spec is meant to represent the overhead required for virt-launcher and libvirt to run within the same environment as the qemu guest.

Looking at the ps aux output, the actual overhead of all the processes (minux qemu) within the Pod is around .21 Gi for libvirt and virt-launcher. That only leaves .02 Gi (or 20Mi) overhead for the qemu process itself.

Looking at the KubeVirt code, I see this being added as overhead to account for shared libraries and processes within the pod that exclude qemu.

overhead.Add(resource.MustParse("138Mi"))
The reality of what I'm seeing from the ps aux screenshot is that the RSS for all the processes in the virt launcher pod is more like 214Mi (or 0.21 Gi), not 138Mi.

Steps to Reproduce:

Not sure

Actual results:

The fact that the customer is hitting the memory limits and the VMI pod is being killed shows that our memory calculations aren't completely accurate. More memory is being consumed then expected.

Expected results:

Based on the memory calculations this shouldn't an issue but the VMI is constantly getting OOM a workaround had to be established in order to stabilize the customer environment. If our overhead calculations here are inaccurate, more customers will get impacted by OOM errors soon.

Additional info:

As a workaround that was established by shift-virtualization to get around this: There's a trick we can give to increase the Pod's overhead to avoid the OOM. This trick is basically the inverse of how we document memory overcommit here.

The trick is to tell the Guest OS that it has less memory than what we're actually requesting for the Pod's environment.

On the domain spec, If we originally set the request/limit to 22 GI like this...

resources:
requests:
cpu: '4'
memory: 22Gi
limits:
cpu: '4'
memory: 22Gi
We'd change that section to look like this...

memory:
guest: 22Gi
resources:
requests:
cpu: '4'
memory: 23Gi
limits:
cpu: '4'
memory: 23Gi

That tells the guest it has only 22Gi of memory, while giving the Pod's environment more memory.

— Additional comment from Courtney Ruhm on 2022-02-09 19:11:06 UTC —

— Additional comment from Courtney Ruhm on 2022-02-09 19:11:38 UTC —

— Additional comment from Courtney Ruhm on 2022-02-09 19:12:06 UTC —

— Additional comment from Courtney Ruhm on 2022-02-09 19:13:30 UTC —

— Additional comment from Artyom on 2022-02-09 20:44:04 UTC —

Moving to CNV. Unsure regarding the correct version for the CNV component.

— Additional comment from David Vossel on 2022-02-10 14:21:03 UTC —

Looking at virt-launcher memory usage over a 24 hour period, I'm not seeing any evidence of a memory leak. This is bug is most likely just a result of our virt-launcher process using more memory than we statically allocate for it in our memory overhead calculations.

— Additional comment from Ken McDonald on 2022-02-10 20:48:31 UTC —

Behavior discovered in the field confirms this issue if a VM definition includes ‘requests’ and ‘limits’ set to equal values. When an RHCOS VM exhausts ‘free’ memory and begins consuming ‘available’ memory, the total amount of memory used by the VM + virt-launcher pod exceeds cgroup limits and the VM is forcibly shutdown with a MEMCG error shown in journalctl

— Additional comment from Kobig on 2022-02-14 10:08:56 UTC —

Not sure but looks like a dup: https://bugzilla.redhat.com/show_bug.cgi?id=2022080

— Additional comment from on 2022-02-16 13:26:28 UTC —

The BZ mentioned in Comment #8 is not a dup as this BZ is about memory consumption.

— Additional comment from Red Hat Bugzilla on 2022-03-01 08:29:13 UTC —

Account disabled by LDAP Audit

— Additional comment from Kedar Bidarkar on 2022-04-14 16:52:20 UTC —

Denys, As this is a customer related bug, we would need an automated testcase here.

— Additional comment from Antonio Cardace on 2022-05-06 08:47:28 UTC —

Fix for this https://github.com/kubevirt/kubevirt/pull/7264 is now in main upstream.

— Additional comment from errata-xmlrpc on 2022-05-09 16:06:14 UTC —

This bug has been added to advisory RHEA-2022:88137 by Antonio Cardace (acardace@redhat.com)

— Additional comment from Denys Shchedrivyi on 2022-05-16 21:04:57 UTC —

(In reply to Kedar Bidarkar from comment #11)
> Denys, As this is a customer related bug, we would need an automated
> testcase here.

As I see it is covered by tier1 test, not sure we can add something

— Additional comment from Denys Shchedrivyi on 2022-05-18 21:53:25 UTC —

I've verified the fix and here my observations:

1. The process virtlogd uses 17Mb of memory, but the limit is 16Mb:

> qemu 19 0.0 0.0 223916 17384 ? Sl 22:15 0:00 /usr/sbin/virtlogd -f /etc/libvirt/virtlogd.conf

it should be fixed by this PR:
https://github.com/kubevirt/kubevirt/pull/7685

2. Memory usage of virt-launcher process increased after 48hrs:

> # VM started:
> qemu 10 0.1 0.2 2002312 60872 ? Sl 22:15 0:04 /usr/bin/virt-launcher --qemu-timeout 253s --name vm-fedora --uid 6b181371-7c41-4783-8f8f-a

> # 12hrs
> qemu 10 0.1 0.2 2002312 67952 ? Sl May16 0:58 /usr/bin/virt-launcher --qemu-timeout 253s --name vm-fedora --uid 6b181371-7c41-4783-8f8f-a

> # 48hrs
> qemu 10 0.1 0.2 2002312 70528 ? Sl May16 3:06 /usr/bin/virt-launcher --qemu-timeout 253s --name vm-fedora --uid 6b181371-7c41-4783-8f8f-a

It still under the limit (75Mb), but I would like to keep VM running for additional several days.

— Additional comment from Denys Shchedrivyi on 2022-05-22 18:11:11 UTC —

Unfortunately, the memory usage of the virt-launcher process is still growing and has exceeded the 75MB limit after 5 days:

> qemu 10 0.1 0.3 2002312 77364 ? Sl May16 9:28 /usr/bin/virt-launcher --qemu-timeout 253s --name vm-fedora --uid 6b181371-7c41-4783-8f8f-a

— Additional comment from Denys Shchedrivyi on 2022-05-25 15:13:52 UTC —

Since the "Overhead Memory consumption calculations" was fixed, moving this bz to Verified state.

For virt-launcher memory leak issue opened new BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2090350

— Additional comment from Denys Shchedrivyi on 2022-05-25 15:14:16 UTC —

Verified on CNV v4.11.0-334