Uploaded image for project: 'AI Platform Core Components'
  1. AI Platform Core Components
  2. AIPCC-3181

Support for llm-d

XMLWordPrintable

    • False
    • Hide

      The Child issues except 1254 are blocked
      Pplx-kernels & deep-ep Legal and license issue NVSHMEM is under a proprietary license, https://docs.nvidia.com/nvshmem/api/sla.html .

      The Git repository lacks both version tags and formal releases, which makes it impossible to pin the project to a specific version with any reliability. 

      Show
      The Child issues except 1254 are blocked Pplx-kernels & deep-ep Legal and license issue NVSHMEM is under a proprietary license,  https://docs.nvidia.com/nvshmem/api/sla.html  . The Git repository lacks both version tags and formal releases, which makes it impossible to pin the project to a specific version with any reliability.  
    • True
    • 0% To Do, 0% In Progress, 100% Done
    • Hide

      Under review. Set to close.

      Show
      Under review. Set to close.

      llm-d [1] is vllm with added layers to distribute work across nodes and accelerators. Red Hat announced the OSS project at Summit 2025.

      The AIPCC team will build new wheels with some additional packages to be used with our vllm in our base images.

      As of writing this there are three known packages that need to be built: nixl, DeepEP, and pplx-kernels.

      The main points of contact are @naisingh@redhat.com, @rhn-support-weaton, @dzonca@redhat.com, and @ldimaggi.

      We can have a single RHAIIS image that contains the llm-d components (nix, DeepEP and pplx-kernels).

      [1] https://llm-d.ai

              fjansen@redhat.com Frank Jansen
              prarit@redhat.com Prarit Bhargava
              Sarah Berry, Vikash Shaw
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: