-
Bug
-
Resolution: Done
-
Critical
-
None
-
None
To Reproduce Steps to reproduce the behavior:
- run ilab data generate with downstream agentic pipeline with vllm with v0.7+
- [Reported by Oleg] vLLM is explicitly throwing an error because we are trying to use our adapters in a way that’s unsupported.To be clear: this has apparently always been an issue that we’ve never caught, but now they are explicitly raising this as an error in versions v0.7+
Expected behavior
- ilab data generate runs successfully with new vllm versions
Device Info (please complete the following information):
-
- ?
Bug impact
- vllm throws an error with ilab sdg downstream pipelines.
- It has been broken in our product pipelines so far without ever complaining with older vllm versions; this also affects classification accuracy of lora adapters.
Known workaround/Potential Fix
- [Oleg]
I have a PR up to fix it here: https://github.com/vllm-project/vllm/pull/14978.
But I just want to flag that we will either need to:
- Build vLLM from the latest version with that commit
- Update the adapter_config.json file in our adapters to have modules_to_save be an empty list and repackage them for the CAR + OCI artifacts
- is blocked by
-
RHELAI-4084 SDG generation phase fails with allowed_token_ids contains out-of-vocab token id error
-
- Verified
-
- links to