Since 4.16.0 pods with memory limits tend to OOM very frequently when writing files larger than memory limit to PVC

4.16.0-rc.4

100% on certain types of storage
(AWS FSx, certain LVMS setups, see additional info)

1. Create pod/pvc that writes a file larger than the container memory limit (attached example)
2.
3.

OOMKilled

Success

Reproducer in OpenShift terms:
https://gist.github.com/akalenyu/949200f48ec89c42429ddb177a2a4dee

The following is relevant for eliminating the OpenShift layer from the issue.
For simplicity, I will focus on BM setup that produces this with LVM storage.
This is also reproducible on AWS clusters with NFS backed NetApp ONTAP FSx.

Further reduced to exclude the OpenShift layer, LVM on a separate (non root) disk:

Prepare disk
lvcreate -T vg1/thin-pool-1 -V 10G -n oom-lv
mkfs.ext4 /dev/vg1/oom-lv 
mkdir /mnt/oom-lv
mount /dev/vg1/oom-lv /mnt/oom-lv

Run container
podman run -m 600m --mount type=bind,source=/mnt/oom-lv,target=/disk --rm -it quay.io/centos/centos:stream9 bash
[root@2ebe895371d2 /]# curl https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-20240527.0.x86_64.qcow2 -o /disk/temp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 47 1157M   47  550M    0     0   111M      0  0:00:10  0:00:04  0:00:06  111MKilled
(Notice the process gets killed, I don't think podman ever whacks the whole container over this though)

The same process on the same hardware on a 4.15 node (9.2) does not produce an OOM
(vs 4.16 which is RHEL 9.4)

For completeness, I will provide some details about the setup behind the LVM pool, though I believe it should not impact the decision about whether this is an issue:
sh-5.1# pvdisplay 
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               vg1
  PV Size               446.62 GiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              114335
  Free PE               11434
  Allocated PE          102901
  PV UUID               <UUID>
Hardware:
SSD (INTEL SSDSC2KG480G8R) behind a RAID 0 of a PERC H330 Mini controller

At the very least, this seems like a change in behavior but tbh I am leaning towards an outright bug.