-
Bug
-
Resolution: Done
-
Major
-
None
-
Quality / Stability / Reliability
-
False
-
-
False
-
CLOSED
-
-
-
Important
-
No
Description of problem:
We have observed an event in which the virt-launcher process was consuming too much memory (+2 GiB) and was oom-killed because the VM has been configured with limits.
Version-Release number of selected component (if applicable):
Openshift 4.10
Openshift Virtualization 4.10.1
registry.redhat.io/container-native-virtualization/virt-launcher@sha256:e5270cccdbbc7c191f69b833d7b813ea20e0a29b0b1adf644ce3630127d4459e
How reproducible:
Only 1 event observed in customer's environment
Steps to Reproduce:
Not reproduced.
Actual results:
The virt-launcher process was killed with this message:
~~~
[414374.527154] virt-launcher invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), order=0, oom_score_adj=997
[414374.537771] CPU: 9 PID: 3541104 Comm: virt-launcher Not tainted 4.18.0-305.45.1.el8_4.x86_64 #1
[414374.547598] Hardware name: REDACTED
[414374.559656] Call Trace:
[414374.562495] dump_stack+0x5c/0x80
[414374.566305] dump_header+0x4a/0x1db
[414374.570305] oom_kill_process.cold.32+0xb/0x10
[414374.575371] out_of_memory+0x1ab/0x4a0
[414374.579654] mem_cgroup_out_of_memory+0xe8/0x100
[414374.584917] try_charge+0x65a/0x690
[414374.588916] mem_cgroup_charge+0xca/0x220
[414374.593490] do_anonymous_page+0x101/0x380
[414374.598170] __handle_mm_fault+0x983/0xca0
[414374.602851] ? __dentry_kill+0x121/0x170
[414374.607334] handle_mm_fault+0xc2/0x1d0
[414374.611723] __do_page_fault+0x1ed/0x4c0
[414374.616210] do_page_fault+0x37/0x130
[414374.620404] ? page_fault+0x8/0x30
[414374.624303] page_fault+0x1e/0x30
[414374.628109] RIP: 0033:0x48e653
[414374.631621] Code: 00 48 81 eb 00 01 00 00 48 81 c7 00 01 00 00 48 81 fb 00 01 00 00 73 82 e9 07 ff ff ff c5 fd ef c0 48 81 fb 00 00 00 02 73 46 <c5> fe 7f 07 c5 fe 7f 47 20 c5 fe 7f 47 40 c5 fe 7f 47 60 48 81 eb
[414374.652720] RSP: 002b:000000c0323e47e0 EFLAGS: 00010202
[414374.658662] RAX: 0000000000000000 RBX: 0000000000004000 RCX: 000000c0873da000
[414374.666738] RDX: 000000000003cf4a RSI: 0000000000000020 RDI: 000000c0873e6000
[414374.674814] RBP: 000000c0323e4828 R08: 000000c0873da000 R09: 00007fb004012b90
[414374.682891] R10: 0000000000000008 R11: 00000000000019f4 R12: 0000000000000007
[414374.690966] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
[414374.699073] memory: usage 3366512kB, limit 3366512kB, failcnt 330
[414374.706075] memory+swap: usage 3366512kB, limit 9007199254740988kB, failcnt 0
[414374.714161] kmem: usage 64316kB, limit 9007199254740988kB, failcnt 0
[414374.721371] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice:
[414374.721393] anon 3376525312
file 540672
kernel_stack 2838528
slab 40289128
percpu 522240
sock 0
shmem 0
file_mapped 0
file_dirty 811008
file_writeback 0
anon_thp 692060160
inactive_anon 2828771328
active_anon 547590144
inactive_file 335872
active_file 0
unevictable 0
slab_reclaimable 12228464
slab_unreclaimable 28060664
pgfault 635118
pgmajfault 0
workingset_refault_anon 0
workingset_refault_file 0
workingset_activate_anon 0
workingset_activate_file 0
workingset_restore_anon 0
workingset_restore_file 0
workingset_nodereclaim 0
pgrefill 0
pgscan 3001
pgsteal 190
pgactivate 134053
pgdeactivate 0
pglazyfree 0
pglazyfreed 0
thp_fault_alloc 297
thp_collapse_alloc 0
[414374.848383] Tasks state (memory values in pages):
[414374.854048] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[414374.864096] [3532309] 0 3532309 35955 615 172032 0 -1000 conmon
[414374.873955] [3532321] 0 3532321 1293486 19745 978944 0 997 virt-launcher
[414374.884503] [3532372] 0 3532372 2449039 594970 5861376 0 997 virt-launcher
[414374.895060] [3532391] 0 3532391 371850 7557 503808 0 997 libvirtd
[414374.905108] [3532392] 0 3532392 36649 3670 315392 0 997 virtlogd
[414374.915142] [3532826] 107 3532826 1110789 229074 7213056 0 997 qemu-kvm
[414374.925199] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice/crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,task=virt-launcher,pid=3532372,uid=0
[414374.977305] Memory cgroup out of memory: Killed process 3532372 (virt-launcher) total-vm:9796156kB, anon-rss:2338832kB, file-rss:41112kB, shmem-rss:0kB, UID:0 pgtables:5724kB oom_score_adj:997
[414375.010625] oom_reaper: reaped process 3532372 (virt-launcher), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
~~~
Expected results:
Additional info:
- external trackers