Uploaded image for project: 'Red Hat Enterprise Linux AI'
  1. Red Hat Enterprise Linux AI
  2. RHELAI-3668

vllm Inference is broken in SDG Downstream Agentic Pipeline

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Approved

      To Reproduce Steps to reproduce the behavior:

      • run ilab data generate with downstream agentic pipeline with vllm with v0.7+
      • [Reported by Oleg] vLLM is explicitly throwing an error because we are trying to use our adapters in a way that’s unsupported.To be clear: this has apparently always been an issue that we’ve never caught, but now they are explicitly raising this as an error in versions v0.7+

      Expected behavior

      • ilab data generate runs successfully with new vllm versions

      Device Info (please complete the following information):

        • ?

      Bug impact

      • vllm throws an error with ilab sdg downstream pipelines. 
      • It has been broken in our product pipelines so far without ever complaining with older vllm versions; this also affects classification accuracy of lora adapters. 

      Known workaround/Potential Fix

      • [Oleg]

      I have a PR up to fix it here: https://github.com/vllm-project/vllm/pull/14978.
      But I just want to flag that we will either need to:

      1. Build vLLM from the latest version with that commit
      2. Update the adapter_config.json file in our adapters to have modules_to_save be an empty list and repackage them for the CAR + OCI artifacts

              osilkin@redhat.com Oleg Silkin
              rh-ee-asaluja Aditi Saluja
              Kamesh Akella Kamesh Akella
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: