Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-25029

[2166889] High memory consumption of virt-launcher process

XMLWordPrintable

    • Important
    • No

      Description of problem:
      We have observed an event in which the virt-launcher process was consuming too much memory (+2 GiB) and was oom-killed because the VM has been configured with limits.

      Version-Release number of selected component (if applicable):
      Openshift 4.10
      Openshift Virtualization 4.10.1
      registry.redhat.io/container-native-virtualization/virt-launcher@sha256:e5270cccdbbc7c191f69b833d7b813ea20e0a29b0b1adf644ce3630127d4459e

      How reproducible:
      Only 1 event observed in customer's environment

      Steps to Reproduce:
      Not reproduced.

      Actual results:
      The virt-launcher process was killed with this message:

      ~~~
      [414374.527154] virt-launcher invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), order=0, oom_score_adj=997
      [414374.537771] CPU: 9 PID: 3541104 Comm: virt-launcher Not tainted 4.18.0-305.45.1.el8_4.x86_64 #1
      [414374.547598] Hardware name: REDACTED
      [414374.559656] Call Trace:
      [414374.562495] dump_stack+0x5c/0x80
      [414374.566305] dump_header+0x4a/0x1db
      [414374.570305] oom_kill_process.cold.32+0xb/0x10
      [414374.575371] out_of_memory+0x1ab/0x4a0
      [414374.579654] mem_cgroup_out_of_memory+0xe8/0x100
      [414374.584917] try_charge+0x65a/0x690
      [414374.588916] mem_cgroup_charge+0xca/0x220
      [414374.593490] do_anonymous_page+0x101/0x380
      [414374.598170] __handle_mm_fault+0x983/0xca0
      [414374.602851] ? __dentry_kill+0x121/0x170
      [414374.607334] handle_mm_fault+0xc2/0x1d0
      [414374.611723] __do_page_fault+0x1ed/0x4c0
      [414374.616210] do_page_fault+0x37/0x130
      [414374.620404] ? page_fault+0x8/0x30
      [414374.624303] page_fault+0x1e/0x30
      [414374.628109] RIP: 0033:0x48e653
      [414374.631621] Code: 00 48 81 eb 00 01 00 00 48 81 c7 00 01 00 00 48 81 fb 00 01 00 00 73 82 e9 07 ff ff ff c5 fd ef c0 48 81 fb 00 00 00 02 73 46 <c5> fe 7f 07 c5 fe 7f 47 20 c5 fe 7f 47 40 c5 fe 7f 47 60 48 81 eb
      [414374.652720] RSP: 002b:000000c0323e47e0 EFLAGS: 00010202
      [414374.658662] RAX: 0000000000000000 RBX: 0000000000004000 RCX: 000000c0873da000
      [414374.666738] RDX: 000000000003cf4a RSI: 0000000000000020 RDI: 000000c0873e6000
      [414374.674814] RBP: 000000c0323e4828 R08: 000000c0873da000 R09: 00007fb004012b90
      [414374.682891] R10: 0000000000000008 R11: 00000000000019f4 R12: 0000000000000007
      [414374.690966] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
      [414374.699073] memory: usage 3366512kB, limit 3366512kB, failcnt 330
      [414374.706075] memory+swap: usage 3366512kB, limit 9007199254740988kB, failcnt 0
      [414374.714161] kmem: usage 64316kB, limit 9007199254740988kB, failcnt 0
      [414374.721371] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice:
      [414374.721393] anon 3376525312
      file 540672
      kernel_stack 2838528
      slab 40289128
      percpu 522240
      sock 0
      shmem 0
      file_mapped 0
      file_dirty 811008
      file_writeback 0
      anon_thp 692060160
      inactive_anon 2828771328
      active_anon 547590144
      inactive_file 335872
      active_file 0
      unevictable 0
      slab_reclaimable 12228464
      slab_unreclaimable 28060664
      pgfault 635118
      pgmajfault 0
      workingset_refault_anon 0
      workingset_refault_file 0
      workingset_activate_anon 0
      workingset_activate_file 0
      workingset_restore_anon 0
      workingset_restore_file 0
      workingset_nodereclaim 0
      pgrefill 0
      pgscan 3001
      pgsteal 190
      pgactivate 134053
      pgdeactivate 0
      pglazyfree 0
      pglazyfreed 0
      thp_fault_alloc 297
      thp_collapse_alloc 0
      [414374.848383] Tasks state (memory values in pages):
      [414374.854048] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
      [414374.864096] [3532309] 0 3532309 35955 615 172032 0 -1000 conmon
      [414374.873955] [3532321] 0 3532321 1293486 19745 978944 0 997 virt-launcher
      [414374.884503] [3532372] 0 3532372 2449039 594970 5861376 0 997 virt-launcher
      [414374.895060] [3532391] 0 3532391 371850 7557 503808 0 997 libvirtd
      [414374.905108] [3532392] 0 3532392 36649 3670 315392 0 997 virtlogd
      [414374.915142] [3532826] 107 3532826 1110789 229074 7213056 0 997 qemu-kvm
      [414374.925199] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice/crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,task=virt-launcher,pid=3532372,uid=0
      [414374.977305] Memory cgroup out of memory: Killed process 3532372 (virt-launcher) total-vm:9796156kB, anon-rss:2338832kB, file-rss:41112kB, shmem-rss:0kB, UID:0 pgtables:5724kB oom_score_adj:997
      [414375.010625] oom_reaper: reaped process 3532372 (virt-launcher), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
      ~~~

      Expected results:

      Additional info:

              sgott@redhat.com Stuart Gott
              rhn-support-jortialc Juan Orti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: