Uploaded image for project: 'AI Platform Core Components'
  1. AI Platform Core Components
  2. AIPCC-10816

Onboard hnswlib into the AIPCC Builder

    • Icon: Story Story
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      Package 'hnswlib' does not build as-is via the AIPCC self-service pipeline and requires builder repository onboarding.

      Build Failure Summary

      Root Cause Analysis: `hnswlib` Build Failure

      Summary

      The build never reached compilation. It failed during the requirements parsing phase due to an invalid version specifier in the requirements file.

      Root Cause

      The file `/collection-repository/collections/torch-2.9.0/cpu-ubi9/requirements.txt` contains the entry:

      hnswlib==any
      

      The string `any` is not a valid PEP 440 version specifier. The Python `packaging` library correctly rejects it:

      packaging._tokenizer.ParserSyntaxError: Expected semicolon (after name with no version specifier) or end
          hnswlib==any
                 ^
      

      The `prepare-requirements` tool calls `Requirement("hnswlib==any")`, which raises an `InvalidRequirement` exception, and the entire bootstrap process aborts at line 0 of the requirements file.

      Fix

      Change the entry in `requirements.txt` from:

      hnswlib==any
      

      to simply:

      hnswlib
      

      An unconstrained package name (with no version specifier) already means "any version is acceptable," which is presumably the intended semantics. If a specific version is needed, use a valid PEP 440 version string (e.g., `hnswlib==0.8.0`).

      Notes

      • This is a requirements file syntax error, not a compilation or packaging issue with `hnswlib` itself.
      • Because the failure occurs in `prepare-requirements` before `fromager` bootstrapping begins, no build artifacts (`build-sequence-summary.md`, logs, etc.) were produced — which matches the `WARNING: no matching files` messages in the artifact upload phase.

      Packaging Analysis Summary

      Here is the executive summary formatted as a JIRA comment in JIRA wiki markup:

      Executive Summary: hnswlib Build Analysis

      hnswlib (v0.8.0) is a C++ header-only approximate nearest neighbor search library with Python bindings via pybind11. It requires source compilation — no pre-built wheels exist on PyPI for any platform or version. The build itself is moderate complexity (4/10): a single C++ source file (

      python_bindings/bindings.cpp

      ) compiled against pybind11 and numpy headers, with no external C/C++ library dependencies. The package is licensed under Apache-2.0, which is fully compatible with Red Hat redistribution requirements.

      Three critical blockers must be resolved before building for enterprise distribution. First, the

      PYBIND11_PLUGIN

      macro used in

      bindings.cpp:910

      was removed in pybind11 3.x, and the pyproject.toml has no upper bound on pybind11 — the build will fail unless pybind11 is pinned to

      <3.0

      or the source is patched to use

      PYBIND11_MODULE

      . Second,

      setup.py:57

      references

      setuptools.distutils.errors.CompileError

      , an import path broken on Python 3.12+ — this must be patched. Third, the default

      -march=native

      compiler flag produces non-portable binaries that crash with "Illegal instruction" on CPUs different from the build host; the environment variable

      HNSWLIB_NO_NATIVE=1

      must be set for all distribution builds.

      The recommended build approach is:

      export HNSWLIB_NO_NATIVE=1
      # Apply patches for pybind11 <3.0 pin and distutils import fix
      pip wheel . --no-build-isolation
      

      For x86_64, the library provides runtime SIMD dispatch (SSE/AVX/AVX-512) via CPUID checks, so portable binaries still achieve optimal performance on capable hardware. For aarch64, no SIMD optimizations exist — the library falls back to scalar code with significantly lower performance. The upstream project is effectively unmaintained: 271 open issues, last release December 2023, and CI only tests up to Python 3.10.

      Key stakeholder note: hnswlib is a CPU-only library with no CUDA or ROCm support. If the onboarding requirement (per AIPCC-10799) includes GPU-accelerated vector search indexes, hnswlib cannot satisfy that requirement. Alternatives such as faiss-gpu or NVIDIA cuVS/RAFT should be evaluated for GPU index targets. hnswlib can only serve the CPU index use case.

              epacific@redhat.com Einat Pacifici
              aipcc-jira-bot@redhat.com AIPCC JIRABOT
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: