Uploaded image for project: 'AI Platform Core Components'
  1. AI Platform Core Components
  2. AIPCC-8724

cuda13.0-ubi9 index for 3.3 has cupy-cuda12x==13.6.0

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • None
    • Accelerator Enablement
    • None
    • AIPCC Accelerators 24
    • Critical

      Description of problem:

      The CUDA 13.0 index comes with a cupy-cuda12x package for CUDA 12.x, https://console.redhat.com/api/pypi/public-rhai/rhoai/3.3/cuda13.0-ubi9/simple/cupy-cuda12x/ .The package does not work with CUDA 13.

      The issue affects its-hub and vLLM for CUDA 13.0

      Version numbers (base image, wheels, builder, etc):

      RHAI 3.3 index

      Steps to Reproduce:

      $ podman run -ti --rm quay.io/aipcc/base-images/cuda-13.0-el9.6:3.3.0-1768412351 
      
      (app-root) /opt/app-root$ uv pip install cupy-cuda13x
      Using Python 3.12.9 environment at: .
        × No solution found when resolving dependencies:
        ╰─▶ Because cupy-cuda13x was not found in the package registry and you require cupy-cuda13x, we can conclude that your requirements are unsatisfiable.
      
      (app-root) /opt/app-root$ uv pip install cupy-cuda12x
      Using Python 3.12.9 environment at: .
      Resolved 3 packages in 2.66s
      Prepared 3 packages in 1.39s
      Installed 3 packages in 606ms
      Bytecode compiled 1155 files in 397ms
       + cupy-cuda12x==13.6.0
       + fastrlock==0.8.3
       + numpy==2.4.1
          

      Actual results:

      $ /opt/app-root/lib/tools/scanlibs.py --silent
      ERROR: Problem detected in /opt/app-root/lib64/python3.12/site-packages/cupy/cuda/cufft.cpython-312-x86_64-linux-gnu.so
      ERROR: Problem detected in /opt/app-root/lib64/python3.12/site-packages/cupy/cuda/jitify.cpython-312-x86_64-linux-gnu.so
      ERROR: Problem detected in /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cublas.cpython-312-x86_64-linux-gnu.so
      ERROR: Problem detected in /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cudnn.cpython-312-x86_64-linux-gnu.so
      ERROR: Problem detected in /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cusolver.cpython-312-x86_64-linux-gnu.so
      ERROR: Problem detected in /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cutensor.cpython-312-x86_64-linux-gnu.so
      ERROR: Problem detected in /opt/app-root/lib64/python3.12/site-packages/cupyx/cusolver.cpython-312-x86_64-linux-gnu.so
      ERROR: 5 libraries with problems:
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy/cuda/cufft.cpython-312-x86_64-linux-gnu.so: missing dependencuies ['libcufft.so.11']
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy/cuda/cufft.cpython-312-x86_64-linux-gnu.so: undefined/missing symbols [('cufftExecD2Z', 'libcufft.so.11'), ('cufftCreate', 'libcufft.so.11'), ('cufftSetStream', 'libcufft.so.11'), ('cufftExecC2C', 'libcufft.so.11'), ('cufftGetVersion', 'libcufft.so.11'), ('cufftExecZ2D', 'libcufft.so.11'), ('cufftMakePlanMany', 'libcufft.so.11'), ('cufftXtSetWorkArea', 'libcufft.so.11'), ('cufftExecZ2Z', 'libcufft.so.11'), ('cufftXtExecDescriptorZ2Z', 'libcufft.so.11')]
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy/cuda/jitify.cpython-312-x86_64-linux-gnu.so: missing dependencuies ['libnvrtc.so.12']
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy/cuda/jitify.cpython-312-x86_64-linux-gnu.so: undefined/missing symbols [('nvrtcCreateProgram', 'libnvrtc.so.12'), ('nvrtcAddNameExpression', 'libnvrtc.so.12'), ('nvrtcGetProgramLog', 'libnvrtc.so.12'), ('nvrtcGetErrorString', 'libnvrtc.so.12'), ('nvrtcGetProgramLogSize', 'libnvrtc.so.12'), ('nvrtcDestroyProgram', 'libnvrtc.so.12'), ('nvrtcCompileProgram', 'libnvrtc.so.12')]
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cublas.cpython-312-x86_64-linux-gnu.so: missing dependencuies ['libcublas.so.12']
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cublas.cpython-312-x86_64-linux-gnu.so: undefined/missing symbols [('cublasDgemmBatched', 'libcublas.so.12'), ('cublasCgetriBatched', 'libcublas.so.12'), ('cublasDtrttp', 'libcublas.so.12'), ('cublasSsyrk_v2', 'libcublas.so.12'), ('cublasDsbmv_v2', 'libcublas.so.12'), ('cublasCdgmm', 'libcublas.so.12'), ('cublasSgemm_v2', 'libcublas.so.12'), ('cublasZsyrk_v2', 'libcublas.so.12'), ('cublasIdamax_v2', 'libcublas.so.12'), ('cublasZtrsm_v2', 'libcublas.so.12')]
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cusolver.cpython-312-x86_64-linux-gnu.so: missing dependencuies ['libcusolver.so.11']
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupy_backends/cuda/libs/cusolver.cpython-312-x86_64-linux-gnu.so: undefined/missing symbols [('cusolverDnSgesvdj_bufferSize', 'libcusolver.so.11'), ('cusolverDnZungqr', 'libcusolver.so.11'), ('cusolverDnSHgels', 'libcusolver.so.11'), ('cusolverDnDHgels_bufferSize', 'libcusolver.so.11'), ('cusolverDnDgesvd', 'libcusolver.so.11'), ('cusolverDnZCgesv_bufferSize', 'libcusolver.so.11'), ('cusolverDnDSgels_bufferSize', 'libcusolver.so.11'), ('cusolverDnSgesvdaStridedBatched_bufferSize', 'libcusolver.so.11'), ('cusolverDnZgesvdaStridedBatched', 'libcusolver.so.11'), ('cusolverDnSXgels_bufferSize', 'libcusolver.so.11')]
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupyx/cusolver.cpython-312-x86_64-linux-gnu.so: missing dependencuies ['libcusolver.so.11']
      ERROR: /opt/app-root/lib64/python3.12/site-packages/cupyx/cusolver.cpython-312-x86_64-linux-gnu.so: undefined/missing symbols [('cusolverDnSorgqr', 'libcusolver.so.11'), ('cusolverDnCungqr', 'libcusolver.so.11'), ('cusolverDnCgesvd', 'libcusolver.so.11'), ('cusolverDnSgesvd', 'libcusolver.so.11'), ('cusolverDnDorgqr', 'libcusolver.so.11'), ('cusolverDnSgeqrf', 'libcusolver.so.11'), ('cusolverDnCgeqrf', 'libcusolver.so.11'), ('cusolverDnZungqr', 'libcusolver.so.11'), ('cusolverDnZgeqrf', 'libcusolver.so.11'), ('cusolverDnZgesvd', 'libcusolver.so.11')]
      problems detected
      

      Expected results:

      The CUDA 13.0 index should NOT contain cupy-cuda12x and instead have cupy-cuda13x. The cupy-cuda12x package must be removed from the index.

      Additional info:

              rh-ee-dtrifiro Daniele Trifirò
              cheimes@redhat.com Christian Heimes
              Christian Heimes, Pavan Kalyan Reddy Cherupally, Reshmi Aravind, Vikash Shaw
              Frank's Team
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: