OOM on x86-03.stream.rdu2.redhat.com

Hi, I have two recent build failures that appear to be caused by out-of-memory condition on the builder x86-03.stream.rdu2.redhat.com:

• https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=3508301
• https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=3508306

First problem I notice is it appears the same builder was running two jobs at once. That makes failure drastically more likely.

Next, looking at the hw_info.log for this builder, I see the builder has 40 vCPUs. A minimum of 80 GB of RAM would be needed to have a good chance of the build not running out of memory, assuming no other build jobs are in progress. Since it's doing two WebKitGTK builds at the same time, that suggests 160 GB minimum. But I see in the hw_info.log that only 50 GB of RAM was available when starting the c9s job, and only 36 GB was available when starting the c8s job.

Please either drastically reduce the number of vCPUs, or drastically increase the RAM. Otherwise, I would have to hardcode a parallelism limit into the spec file, which would be sad and make things much slower. Having an appropriate amount of RAM per vCPU allows builds to take full advantage of available vCPUs; having more than can be supported by the RAM is not useful. (In c10s we have a new %limit_build macro that allows us to limit build parallelism based on the amount of RAM available. However, this macro does not exist in c9s or c8s.)

Please also investigate why two jobs were running on the same builder at the same time. Surely this should not be permitted, at least not for WebKitGTK or other heavy packages (Firefox, Inkscape, LibreOffice, LLVM).