1. Proposed title of this feature request

Support Topology-Aware Gang Scheduling

2. What is the nature and description of the request?

Add support for topology-aware scheduling to OpenShift, which will allow llm-d to optimize the placement of vLLM pods, reducing the cost of inference.

Distributed LLM inference performance involves many kinds of GPU-to-GPU communication and as such  is highly sensitive to the physical placement of pods on the underlying hardware.

Tensor parallelism relies on dense AllReduce collective communication operations provided by libraries like NCCL or RCCL. These AllReduces, typically used in single-node inference deployments, are faster if all worker GPUs reside within a PCIe domain.

WideEP deployments for multinode mixture-of-expert models use sparse all-to-all dispatch and combine operations like those in DeepEP. These All2All ops are one of the most expensive costs of inference at scale and are faster if traffic between racks is minimized. This is particularly true for multi-node NVLINK (MNNVL) systems like the GB200 NVL72. Additionally gang scheduling provides simultaneous scheduling of all pods in a deployment that may involve a dozen or more nodes, requiring the composability of both gang-scheduling and topology-aware scheduling.

Prefill-decode (P/D) disaggregation is a technique that splits the phases of inference into separate deployments. This allows specialization of how the prefill and decode phases are parallelized, and can also drastically reduce P99 inter-token latency. P/D disaggregation relies on libraries like NIXL to transfer KV cache state between inference pods using RDMA over interconnects like Infiniband, RoCE, or NVLINK. Placing a prefill and decode instance on the same node allows the use of the faster NVLINK interconnect for transfers between them, whereas cross-node KV transfers may use the slower Infiniband or RoCE.

Topology-aware scheduling of vLLM pods will allow llm-d to optimize communication operations, which is critically important for cluster-scale inference.

3. Why does the customer need this? (List the business requirements here)

This feature will support the core functionality of llm-d – optimizing the cost of cluster-scale inference.

4. List any affected packages or components.
llm-d