Uploaded image for project: 'AI Platform Core Components'
  1. AI Platform Core Components
  2. AIPCC-10344

Neuron container build fails with a ldd warning and erroring out for neuroncc

    • False
    • Hide

      None

      Show
      None
    • False
    • AIPCC Accelerators 26

      Description of problem:

          Neuron build fails with the following error - 
      
      ldd: warning: you do not have execution permission for `/opt/app-root/lib64/python3.12/site-packages/neuronxcc/data/custom_op/libbuiltincustomop_cpu6.stripped.so'
      	not a dynamic executable
      Traceback (most recent call last):
        File "/opt/app-root/lib/tools/scanlibs.py", line 176, in <module>
          main()
        File "/opt/app-root/lib/tools/scanlibs.py", line 145, in main
          missing_libs, undef_symbols = ldd(lib)
                                        ^^^^^^^^
        File "/opt/app-root/lib/tools/scanlibs.py", line 60, in ldd
          out = subprocess.check_output(
                ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.12/subprocess.py", line 468, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.12/subprocess.py", line 573, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ldd', '--function-relocs', '/opt/app-root/lib64/python3.12/site-packages/neuronxcc/data/custom_op/libbuiltincustomop_cpu6.stripped.so']' returned non-zero exit status 1.
      subprocess exited with status 1
      subprocess exited with status 1
      Error: building at STEP "RUN ${APP_ROOT}/lib/tools/fromager-rpm-check.py --check-rpmdb     && ${APP_ROOT}/lib/tools/scanlibs.py --ignore '_XLAC.cpython-312-x86_64-linux-gnu.so'     && ${APP_ROOT}/lib/tools/selftest.py     && which vllm": exit status 1
      

      Version numbers (base image, wheels, builder, etc):

          Neuron base image

      Steps to Reproduce:

          1. MR - https://gitlab.com/redhat/rhel-ai/rhaiis/containers/-/merge_requests/367     
          2. Start pipeline. 
          3. Neuron build fails with the error below.
          

      Actual results:

        ldd: warning: you do not have execution permission for `/opt/app-root/lib64/python3.12/site-packages/neuronxcc/data/custom_op/libbuiltincustomop_cpu6.stripped.so'
      	not a dynamic executable
      Traceback (most recent call last):
        File "/opt/app-root/lib/tools/scanlibs.py", line 176, in <module>
          main()
        File "/opt/app-root/lib/tools/scanlibs.py", line 145, in main
          missing_libs, undef_symbols = ldd(lib)
                                        ^^^^^^^^
        File "/opt/app-root/lib/tools/scanlibs.py", line 60, in ldd
          out = subprocess.check_output(
                ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.12/subprocess.py", line 468, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.12/subprocess.py", line 573, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ldd', '--function-relocs', '/opt/app-root/lib64/python3.12/site-packages/neuronxcc/data/custom_op/libbuiltincustomop_cpu6.stripped.so']' returned non-zero exit status 1.
      subprocess exited with status 1
      subprocess exited with status 1
      Error: building at STEP "RUN ${APP_ROOT}/lib/tools/fromager-rpm-check.py --check-rpmdb     && ${APP_ROOT}/lib/tools/scanlibs.py --ignore '_XLAC.cpython-312-x86_64-linux-gnu.so'     && ${APP_ROOT}/lib/tools/selftest.py     && which vllm": exit status 1
      Expected results:
      
      
      

      Additional info:

              emacchi@redhat.com Emilien Macchi
              rh-ee-debroy Debolina Roy
              Frank's Team
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: