There is no demand for this and it's being kept open for long time. Please reopen if needed.

Agreed. Let's close this. We can always reopen if needed.

mprivozn@redhat.com - I see no activity on this for an extended period of time - are you ok with closing this as WONTFIX?  Additionally if it's still important perhaps create an upstream tracker.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

(In reply to Michal Privoznik from comment #22)
> (In reply to David Hildenbrand from comment #21)
>
> Spoiler alert: I know next to nothing about memory mgmt.
>
> > It's still sub-optimal, though. Hotplugging a 128 MiB DIMM first followed by
> > a 256 MiB DIMM would unnecessarily create a hole ...
>
> Can you enlighten me please - why are holes bad? Is it because if a DIMM is
> backed by a hugepage then it's wasteful?

Because the GPA will be fragmented. For Linux, this implies that certain operations, such as memory compaction, get more expensive because Linux as to consider holes in memory zones and has to scan over these holes.

Further, Linux cannot make use of that memory for larger allocations (such as gigantic pages). It's a secondary concern, though.

> Also - how is this solved at real HW level? I mean, when I plug a DIMM into
> a slot, it might too create a hole, couldn't it?

I was told by Intel a while ago that real HW does not support hotplug of individual DIMMs, but only complete NUMA nodes. Holes between other nodes are less of a concern (in Linux, it's separate memory zones either way). So it's not really an issue on real HW.

(In reply to David Hildenbrand from comment #21)

Spoiler alert: I know next to nothing about memory mgmt.

> It's still sub-optimal, though. Hotplugging a 128 MiB DIMM first followed by
> a 256 MiB DIMM would unnecessarily create a hole ...

Can you enlighten me please - why are holes bad? Is it because if a DIMM is backed by a hugepage then it's wasteful?
Also - how is this solved at real HW level? I mean, when I plug a DIMM into a slot, it might too create a hole, couldn't it?

(In reply to David Hildenbrand from comment #20)
> (In reply to Michal Privoznik from comment #19)
> > (In reply to David Hildenbrand from comment #17)
> > > Getting that intended minimum alignment from the user is IMHO better than
> > > hard-coding it in QEMU and having to deal with compat handling.
> >
> > But problem is whether user will know what value to put in. To sum up:
> >
> > QEMU knows what values are acceptable, but not which OS is running in the
> > guest,
> > libvirt does not know what value to pass, nor which OS is running in the
> > guest,
> > user does not know what value to pass, but it knows what OS is running in
> > the guest.
>
> QEMU most certainly knows the least
>
> Again, the user already has to be aware of guest OS restrictions. While
> hotplugging a 128 MiB DIMM to a VM running an arm64 Linux kernel with 4k
> page size will work, it's unusable by an arm64 Linux kernel with a 64k page
> size. Just like the minimum granularity, the alignment is guest-OS specific.
>
> >
> > So I wonder whether we should:
> > a) chose a reasonable default in QEMU, and possibly
>
> I'm afraid that will require compat machine changes.
>
> And there is no reasonable default for arm64, for example, without knowing
> what's running inside the VM. Using an alignment of 512MiB just because the
> guest could be running a 64k kernel fragments guest physical address space
> when hotplugging 128 MiB DIMMs.

BTW, I was playing with the idea of deciding the alignment based on the size.

DIMM size is multiples of 128 MiB -> align to 128 MiB
DIMM size is multiples of 256 MiB -> align to 256 MiB
DIMM size is multiples of 512 MiB -> align to 512 MiB

It's still sub-optimal, though. Hotplugging a 128 MiB DIMM first followed by a 256 MiB DIMM would unnecessarily create a hole ...

(In reply to Michal Privoznik from comment #19)
> (In reply to David Hildenbrand from comment #17)
> > Getting that intended minimum alignment from the user is IMHO better than
> > hard-coding it in QEMU and having to deal with compat handling.
>
> But problem is whether user will know what value to put in. To sum up:
>
> QEMU knows what values are acceptable, but not which OS is running in the
> guest,
> libvirt does not know what value to pass, nor which OS is running in the
> guest,
> user does not know what value to pass, but it knows what OS is running in
> the guest.

QEMU most certainly knows the least

Again, the user already has to be aware of guest OS restrictions. While hotplugging a 128 MiB DIMM to a VM running an arm64 Linux kernel with 4k page size will work, it's unusable by an arm64 Linux kernel with a 64k page size. Just like the minimum granularity, the alignment is guest-OS specific.

>
> So I wonder whether we should:
> a) chose a reasonable default in QEMU, and possibly

I'm afraid that will require compat machine changes.

And there is no reasonable default for arm64, for example, without knowing what's running inside the VM. Using an alignment of 512MiB just because the guest could be running a 64k kernel fragments guest physical address space when hotplugging 128 MiB DIMMs.

(In reply to David Hildenbrand from comment #17)
> Getting that intended minimum alignment from the user is IMHO better than
> hard-coding it in QEMU and having to deal with compat handling.

But problem is whether user will know what value to put in. To sum up:

QEMU knows what values are acceptable, but not which OS is running in the guest,
libvirt does not know what value to pass, nor which OS is running in the guest,
user does not know what value to pass, but it knows what OS is running in the guest.

So I wonder whether we should:
a) chose a reasonable default in QEMU, and possibly
b) offer users a way to tweak the alignment.

(In reply to Igor Mammedov from comment #16)
> QEMU already reserves 1G of GPA per device, so why not align every one on 1G
> border (without adding any new options)?

We only do that on x86 so far IIRC, and only for memory devices that require an ACPI slot (we don't know how many other devices we might have). The underlying reason IIRC, was to handle memory backends with gigantic pages that require a certain alignment in GPA. So on x86 we could eventually align only such devices (DIMMs/NVDIMMs) to 1 GiB without further changes. For everything else, we could break existing setups eventually and would require some compat handling (I recall that any such gpa layout changes might require compat handling, but at least libvirt should be able to deal with that). A user option won't require gluing that to compat machines.

Aligning all DIMMs to 1 GiB is also not really desired IMHO. If you hotplug multiple smaller DIMMs (< 1 GiB, which apparently users do for Kata and such), you'd get quite a lot of (large) GPA holes in between, implying that PFN walkers (like compaction) inside the VM get more expensive (i.e., zones not contiguous) and that such memory can never get used for larger contiguous allocations (such as gigantic pages).

Ideally, we don't get any holes, even when hotplugging DIMMs that are any multiples of 128 MiB (on x86), which is the common case and only doesn't work because NVDIMMs do weird stuff with the labels. But that 128 MiB alignment is both guest and arch specific.

Getting that intended minimum alignment from the user is IMHO better than hard-coding it in QEMU and having to deal with compat handling.