Loading...

XML

Word

Printable

Type: Bug
Resolution: Duplicate
Priority: Major
Fix Version/s: CNV v4.14.5
Affects Version/s: None
Component/s: Storage Platform
Labels:
None

Story Points:
0.42
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
[QE] How to address?:
---
[QE] Why QE missed?:
---
Market:

Sprint:
Storage Core Sprint 251

Regression:
No

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

Very similar to CNV-28628, but here it is the 'dd' command to zero (preallocate) the disk that triggers the oom-kill, not the qemu-img conversion of the above bug. So its not just qemu-img that needs to be improved, this other path also can hit it.

Version-Release number of selected component (if applicable):

OCP 4.14.8
CNV 4.14.2
LVMS 4.14.1

How reproducible:

Always

Steps to Reproduce:

1. Create an empty disk

$ oc get dv -o yaml fedora-39-vms-1
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  annotations:
    cdi.kubevirt.io/storage.usePopulator: "true"
  creationTimestamp: "2024-01-22T05:09:37Z"
  generation: 1
  labels:
    kubevirt.io/created-by: e9b20cd5-3d37-4c4d-ae13-a0dd6f9ed968
  name: fedora-39-vms-1
  namespace: homelab
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: fedora-39
    uid: e9b20cd5-3d37-4c4d-ae13-a0dd6f9ed968
  resourceVersion: "19150076"
  uid: 413ffd0d-6560-4606-b8c8-b8035d181893
spec:
  preallocation: true
  source:
    blank: {}
  storage:
    resources:
      requests:
        storage: 150Gi
    storageClassName: lvms-ssd

2. Check it "preallocating" in 'ps' in the node:

107       702525  702513  1 05:24 ?        00:00:00 /usr/bin/virt-cdi-importer -alsologtostderr -v=1
107       702564  702525 73 05:24 ?        00:00:08 /usr/bin/coreutils --coreutils-prog-shebang=dd /usr/bin/dd if=/dev/zero of=/dev/cdi-block-volume bs=1048576 count=153600 seek=0 oflag=seek_bytes

3. Wait for it to go over the cgroupv1 memory limit

Jan 22 05:25:38 white.shift.home.arpa kernel: dd invoked oom-killer: gfp_mask=0x101c00(GFP_NOIO|__GFP_HARDWALL|__GFP_WRITE), order=0, oom_score_adj=999
Jan 22 05:25:38 white.shift.home.arpa kernel: CPU: 10 PID: 702564 Comm: dd Not tainted 5.14.0-284.48.1.el9_2.x86_64 #1
Jan 22 05:25:38 white.shift.home.arpa kernel: Hardware name: Intel(R) Client Systems NUC9i7QNX/NUC9i7QNB, BIOS QXCFL579.0071.2022.1130.1331 11/30/2022
Jan 22 05:25:38 white.shift.home.arpa kernel: Call Trace:
Jan 22 05:25:38 white.shift.home.arpa kernel:  <TASK>
Jan 22 05:25:38 white.shift.home.arpa kernel:  dump_stack_lvl+0x34/0x48
Jan 22 05:25:38 white.shift.home.arpa kernel:  dump_header+0x4a/0x201
Jan 22 05:25:38 white.shift.home.arpa kernel:  oom_kill_process.cold+0xb/0x10
Jan 22 05:25:38 white.shift.home.arpa kernel:  out_of_memory+0xed/0x2e0
Jan 22 05:25:38 white.shift.home.arpa kernel:  mem_cgroup_out_of_memory+0x13a/0x150
Jan 22 05:25:38 white.shift.home.arpa kernel:  try_charge_memcg+0x6df/0x7a0
Jan 22 05:25:38 white.shift.home.arpa kernel:  charge_memcg+0x7a/0xf0
Jan 22 05:25:38 white.shift.home.arpa kernel:  __mem_cgroup_charge+0x29/0x80
Jan 22 05:25:38 white.shift.home.arpa kernel:  __filemap_add_folio+0x224/0x5a0
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? scan_shadow_nodes+0x30/0x30
Jan 22 05:25:38 white.shift.home.arpa kernel:  filemap_add_folio+0x37/0xa0
Jan 22 05:25:38 white.shift.home.arpa kernel:  __filemap_get_folio+0x1e6/0x330
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? blkdev_write_begin+0x20/0x20
Jan 22 05:25:38 white.shift.home.arpa kernel:  pagecache_get_page+0x15/0x90
Jan 22 05:25:38 white.shift.home.arpa kernel:  block_write_begin+0x24/0xf0
Jan 22 05:25:38 white.shift.home.arpa kernel:  generic_perform_write+0xbe/0x200
Jan 22 05:25:38 white.shift.home.arpa kernel:  __generic_file_write_iter+0xe5/0x1a0
Jan 22 05:25:38 white.shift.home.arpa kernel:  blkdev_write_iter+0xe4/0x160
Jan 22 05:25:38 white.shift.home.arpa kernel:  new_sync_write+0xfc/0x190
Jan 22 05:25:38 white.shift.home.arpa kernel:  vfs_write+0x1ef/0x280
Jan 22 05:25:38 white.shift.home.arpa kernel:  ksys_write+0x5f/0xe0
Jan 22 05:25:38 white.shift.home.arpa kernel:  do_syscall_64+0x59/0x90
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? do_syscall_64+0x69/0x90
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? do_syscall_64+0x69/0x90
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? syscall_exit_to_user_mode+0x22/0x40
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? do_syscall_64+0x69/0x90
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? do_syscall_64+0x69/0x90
Jan 22 05:25:38 white.shift.home.arpa kernel:  ? sysvec_apic_timer_interrupt+0x3c/0x90
Jan 22 05:25:38 white.shift.home.arpa kernel:  entry_SYSCALL_64_after_hwframe+0x64/0xce
Jan 22 05:25:38 white.shift.home.arpa kernel: RIP: 0033:0x7f122248d917
Jan 22 05:25:38 white.shift.home.arpa kernel: Code: 0b 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
Jan 22 05:25:38 white.shift.home.arpa kernel: RSP: 002b:00007ffc7ed0ae88 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Jan 22 05:25:38 white.shift.home.arpa kernel: RAX: ffffffffffffffda RBX: 0000000000100000 RCX: 00007f122248d917
Jan 22 05:25:38 white.shift.home.arpa kernel: RDX: 0000000000100000 RSI: 00007f12221ac000 RDI: 0000000000000001
Jan 22 05:25:38 white.shift.home.arpa kernel: RBP: 00007f12221ac000 R08: 00007f12221ac000 R09: 0000000000000000
Jan 22 05:25:38 white.shift.home.arpa kernel: R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000000000
Jan 22 05:25:38 white.shift.home.arpa kernel: R13: 0000000000100000 R14: 00007f12221ac000 R15: 0000000000000000
Jan 22 05:25:38 white.shift.home.arpa kernel:  </TASK>
Jan 22 05:25:38 white.shift.home.arpa kernel: memory: usage 585936kB, limit 585936kB, failcnt 57839141
Jan 22 05:25:38 white.shift.home.arpa kernel: memory+swap: usage 585936kB, limit 9007199254740988kB, failcnt 0
Jan 22 05:25:38 white.shift.home.arpa kernel: kmem: usage 28024kB, limit 9007199254740988kB, failcnt 0
Jan 22 05:25:38 white.shift.home.arpa kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede83558_db4b_4554_b8f0_faf68fc52580.slice:
Jan 22 05:25:38 white.shift.home.arpa kernel: anon 24018944
                                              file 547282944
                                              kernel 28696576
                                              kernel_stack 294912
                                              pagetables 483328
                                              percpu 72744
                                              sock 0
                                              vmalloc 28672
                                              shmem 106496
                                              zswap 0
                                              zswapped 0
                                              file_mapped 0
                                              file_dirty 0
                                              file_writeback 547176448
                                              swapcached 0
                                              anon_thp 4194304
                                              file_thp 0
                                              shmem_thp 0
                                              inactive_anon 24027136
                                              active_anon 98304
                                              inactive_file 273567744
                                              active_file 273567744
                                              unevictable 0
                                              slab_reclaimable 27394320
                                              slab_unreclaimable 342720
                                              slab 27737040
                                              workingset_refault_anon 0
                                              workingset_refault_file 7
                                              workingset_activate_anon 0
                                              workingset_activate_file 0
                                              workingset_restore_anon 0
                                              workingset_restore_file 0
                                              workingset_nodereclaim 900708
                                              pgscan 719480750
                                              pgsteal 58830618
                                              pgscan_kswapd 0
                                              pgscan_direct 719480750
                                              pgsteal_kswapd 0
                                              pgsteal_direct 58830618
                                              pgfault 63587
                                              pgmajfault 0
                                              pgrefill 630525186
                                              pgactivate 660529076
                                              pgdeactivate 630525177
                                              pglazyfree 0
                                              pglazyfreed 0
                                              zswpin 0
                                              zswpout 0
                                              thp_fault_alloc 54
                                              thp_collapse_alloc 0
Jan 22 05:25:38 white.shift.home.arpa kernel: Tasks state (memory values in pages):
Jan 22 05:25:38 white.shift.home.arpa kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 22 05:25:38 white.shift.home.arpa kernel: [ 702513]     0 702513     2076      485    53248        0         -1000 conmon
Jan 22 05:25:38 white.shift.home.arpa kernel: [ 702525]   107 702525   474982    15313   409600        0           999 virt-cdi-import
Jan 22 05:25:38 white.shift.home.arpa kernel: [ 702564]   107 702564     1492      864    53248        0           999 dd
Jan 22 05:25:38 white.shift.home.arpa kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=crio-eca8ef2038ca0fb25e65ba754385fd68e53abaadb7baa5875483a2902ab66ade.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede83558_db4b_4554_b8f0_faf68fc52580.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede83558_db4b_4554_b8f0_faf68fc52580.slice/crio-eca8ef2038ca0fb25e65ba754385fd68e53abaadb7baa5875483a2902ab66ade.scope,task=virt-cdi-import,pid=702525,uid=107
Jan 22 05:25:38 white.shift.home.arpa kernel: Memory cgroup out of memory: Killed process 702525 (virt-cdi-import) total-vm:1899928kB, anon-rss:20180kB, file-rss:41072kB, shmem-rss:0kB, UID:107 pgtables:400kB oom_score_adj:999

Actual results:

Takes ages to provision an empty disk, multiple attemps

Expected results:

Bypass page cache when writing zeroes using oflag option.

Additional info:

relates to

CNV-28628 [4.15] virt-cdi-import often killed by oom-killer during VM provisioning

Closed

Assignee:: Alex Kalenyuk

Reporter:: Germano Veit Michel

QA Contact:: Dalia Frank

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Created:: 2024/01/22 5:33 AM

Updated:: 2024/03/24 5:19 PM

Resolved:: 2024/03/24 5:19 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates