-
Story
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
-
False
-
-
False
-
-
Add package 'nemo-toolkit' into the RHAI pipeline onboarding collection.
The package requires builder repository onboarding before it can be added to the RHAI pipeline. This ticket is blocked by the builder onboarding ticket.
Summary
Here is the executive summary formatted as a JIRA comment in JIRA wiki markup:
Executive Summary: nemo-toolkit (NVIDIA NeMo Framework) Packaging Analysis
The nemo-toolkit package (v2.6.2) is a moderate complexity pure-Python package distributed as a py3-none-any wheel, meaning no native compilation is required at build time. The package is Apache-2.0 licensed, which is fully compatible with Red Hat distribution policies. However, a C++ pybind11 helper module is compiled at runtime when blendable dataset functionality is used, requiring g++ and pybind11 headers on the target system. The primary build system is setuptools (PEP 517/518 compliant), and the source is publicly available at GitHub.
The most critical blocker for onboarding is the dependency tree. The package nv_one_logger_pytorch_lightning_integration (required by the [lightning], [core], and [all] extras) is not available on public PyPI and must be sourced from NVIDIA's internal index or patched out. Additionally, numba-cuda is pinned to an exact old version (==0.15.1) with known open compatibility issues (#15331, #15155), and both numba-cuda and cuda-bindings are hard CUDA-only dependencies that must be removed or conditioned for CPU-only builds.
NeMo has zero ROCm/HIP support — there are no ROCm code paths in the codebase, no ROCm CI testing, and no upstream plans to add it. This is the most significant risk for AIPCC-10792. Building for a ROCm index is not feasible without substantial patching of CUDA-specific kernel code (numba-cuda JIT kernels for RNNT loss, spec augmentation). For the CUDA index, the package is a natural fit provided dependency conflicts are resolved. For CPU-only, patching to remove CUDA dependencies is required.
Recommended Actions
- Dependency patching: Remove or replace nv_one_logger_pytorch_lightning_integration from requirements_lightning.txt; consider relaxing the numba-cuda==0.15.1 pin; conditionally remove CUDA-only deps for CPU builds
- Build approach: Source build from sdist or use the existing py3-none-any wheel directly — no platform-specific compilation needed at build time
- ROCm decision: Exclude ROCm from delivery scope or escalate to NVIDIA upstream for ROCm support
- Fallback: For ASR-only use cases, consider speechbrain (MIT-licensed, simpler dependency tree) as an alternative if NeMo proves infeasible
- Environment variables: Set STRICT_NUMBA_COMPAT_CHECK=false and LHOTSE_REQUIRE_TORCHAUDIO=0 for builds; ensure system packages libsndfile1, libsox-fmt-all, ffmpeg are available for full runtime functionality
- blocks
-
AIPCC-10811 Onboard nemo-toolkit into the AIPCC Builder
-
- Closed
-
- mentioned on