Uploaded image for project: 'Red Hat Enterprise Linux AI'
  1. Red Hat Enterprise Linux AI
  2. RHELAI-2418

RHEL AI 1.3 Gaudi3 : ilab config init should create config with `gpus` set to 1

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Release Notes
    • Impediment
    • Known Issue
    • Important
    • Approved

      To Reproduce Steps to reproduce the behavior:

      1. boot system with registry.stage.redhat.io/rhelai1/bootc-intel-rhel9:1.3-1732661719
      2. run ilab config init
      3. Check the config.yaml
      4. [root@dhcp-10-111-212-61 devcloud]# cat .config/instructlab/config.yaml  | grep gpus
          gpus: 8
              gpus: 8
            gpus: 8

        If the config remains like this, `ilab model serve` will fail with:

      5. AssertionError: GPUExecutor only supports single GPU.
        Exception ignored in: <function HabanaExecutor.__del__ at 0x7f37f90eab60>
        Traceback (most recent call last):
          File "/opt/app-root/lib64/python3.11/site-packages/vllm/executor/habana_executor.py", line 197, in __del__
          File "/opt/app-root/lib64/python3.11/site-packages/vllm/executor/habana_executor.py", line 194, in shutdown
        AttributeError: 'HabanaExecutorAsync' object has no attribute 'driver_worker'

      As per cheimes@redhat.com the config should be created with gpus set to 1 ( slack thread  )

      Also by updating the gpus and tensor values in the serve section to 1, the serve and chat works just fine:

      [root@dhcp-10-111-212-61 devcloud]# ILAB_HOME=/var/home/devcloud ilab chat
      ╭────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────╮
      │ Welcome to InstructLab Chat w/ GRANITE-7B-REDHAT-LAB (type /h for help)                                                                                                                    │
      ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
      >>> hello                                                                                                                                                                         [S][default]
      ╭────────────────────────────────────────────────────────────────────────────────── granite-7b-redhat-lab ───────────────────────────────────────────────────────────────────────────────────╮
      │ Hello! How can I assist you today?                                                                                                                                                         │
      ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 0.207 seconds ─╯
      >>> are you alive ?                                                                                                                                                               [S][default]
      ╭────────────────────────────────────────────────────────────────────────────────── granite-7b-redhat-lab ───────────────────────────────────────────────────────────────────────────────────╮
      │ Yes, I am an AI language model designed to help answer your questions and provide information as best I can.                                                                               │
      ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 0.513 seconds ─╯
      >>>      

      Expected behavior

      • <your text here>

      Screenshots

      • Attached Image 

      Device Info (please complete the following information):

      • Hardware Specs: [e.g. Apple M2 Pro Chip, 16 GB Memory, etc.]
      • OS Version: [e.g. Mac OS 14.4.1, Fedora Linux 40]
      • Python Version: [output of \\{{{}python --version{}}}]
      • InstructLab Version: [output of \\{{{}ilab --version{}}}]

      Additional context

      • <your text here>
      • ...

              aliryan Alina Ryan
              cvultur@redhat.com Constantin Daniel Vultur
              Christian Heimes
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: