Package 'nemo-toolkit' does not build as-is via the AIPCC self-service pipeline and requires builder repository onboarding.

Build Failure Summary

Root Cause Analysis: `nemo-toolkit` Build Failure

Problem

The build failed before any compilation or wheel-building began, during the requirements preparation phase. The failure is caused by an invalid version specifier in the requirements file.

The offending line in `/collection-repository/collections/torch-2.9.0/cpu-ubi9/requirements.txt`:

nemo-toolkit==any

Why It Fails

The version specifier `==any` is not valid per PEP 440. The `packaging` library's requirement parser expects either:

• A valid version number after `==` (e.g., `==2.0.0`)
• No version specifier at all (just `nemo-toolkit`)
• A wildcard match like `==*` (though even this is non-standard)

The parser error confirms this:

packaging._tokenizer.ParserSyntaxError: Expected semicolon (after name with no version specifier) or end
    nemo-toolkit==any
                ^

The `prepare-requirements` tool (`package_plugins.cli.prepare_requirements_constraints`) calls `Requirement(line)` from the `packaging` library, which correctly rejects `==any` as syntactically invalid.

Fix

Replace the invalid version specifier in the requirements file. The correct replacement depends on intent:

• If any version is acceptable: Use just the package name with no version constraint:
  ```
  nemo-toolkit
  ```

• If a specific version is needed: Specify a valid PEP 440 version:
  ```
  nemo-toolkit==2.3.0
  ```

• If a version range is needed: Use standard specifiers:
  ```
  nemo-toolkit>=2.0.0
  ```

Summary

This is a configuration error in the collection's requirements file, not a build or compilation issue. The string `==any` is not a recognized version specifier in PEP 440. No code changes to `nemo-toolkit` itself or the build tooling are needed — only the requirements entry needs to be corrected.

Packaging Analysis Summary

Here is the executive summary formatted as a JIRA comment in JIRA wiki markup:

Executive Summary: nemo-toolkit (NVIDIA NeMo Framework) Packaging Analysis

The nemo-toolkit package (v2.6.2) is a moderate complexity pure-Python package distributed as a py3-none-any wheel, meaning no native compilation is required at build time. The package is Apache-2.0 licensed, which is fully compatible with Red Hat distribution policies. However, a C++ pybind11 helper module is compiled at runtime when blendable dataset functionality is used, requiring g++ and pybind11 headers on the target system. The primary build system is setuptools (PEP 517/518 compliant), and the source is publicly available at GitHub.

The most critical blocker for onboarding is the dependency tree. The package nv_one_logger_pytorch_lightning_integration (required by the [lightning], [core], and [all] extras) is not available on public PyPI and must be sourced from NVIDIA's internal index or patched out. Additionally, numba-cuda is pinned to an exact old version (==0.15.1) with known open compatibility issues (#15331, #15155), and both numba-cuda and cuda-bindings are hard CUDA-only dependencies that must be removed or conditioned for CPU-only builds.

NeMo has zero ROCm/HIP support — there are no ROCm code paths in the codebase, no ROCm CI testing, and no upstream plans to add it. This is the most significant risk for AIPCC-10792. Building for a ROCm index is not feasible without substantial patching of CUDA-specific kernel code (numba-cuda JIT kernels for RNNT loss, spec augmentation). For the CUDA index, the package is a natural fit provided dependency conflicts are resolved. For CPU-only, patching to remove CUDA dependencies is required.

Recommended Actions

• Dependency patching: Remove or replace nv_one_logger_pytorch_lightning_integration from requirements_lightning.txt; consider relaxing the numba-cuda==0.15.1 pin; conditionally remove CUDA-only deps for CPU builds
• Build approach: Source build from sdist or use the existing py3-none-any wheel directly — no platform-specific compilation needed at build time
• ROCm decision: Exclude ROCm from delivery scope or escalate to NVIDIA upstream for ROCm support
• Fallback: For ASR-only use cases, consider speechbrain (MIT-licensed, simpler dependency tree) as an alternative if NeMo proves infeasible
• Environment variables: Set STRICT_NUMBA_COMPAT_CHECK=false and LHOTSE_REQUIRE_TORCHAUDIO=0 for builds; ensure system packages libsndfile1, libsox-fmt-all, ffmpeg are available for full runtime functionality