Uploaded image for project: 'Red Hat Enterprise Linux AI'
  1. Red Hat Enterprise Linux AI
  2. RHELAI-4327

Warnings with possible performance degrdation in Ilab model serve

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • rhelai-1.5.1, rhelai-1.5.2
    • Model Production
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Moderate

      On a AMD GPU Azure:
      image: registry.stage.redhat.io/rhelai1/bootc-azure-amd-rhel9:1.5.1-1749212837
      GPU: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300X VF]     
      AZURE Cloud:

      Model

      models/granite-3.1-8b-lab-v2.1 2025-06-06 20:27:45 31.2 GB /var/home/azureuser/.cache/instructlab/models/granite-3.1-8b-lab-v2.1
      models/granite-3.1-8b-starter-v2.1 2025-06-06 20:34:27 31.2 GB /var/home/azureuser/.cache/instructlab/models/granite-3.1-8b-starter-v2.1
      models/mixtral-8x7b-instruct-v0-1 2025-06-06 20:51:45 87.0 GB /var/home/azureuser/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
      models/prometheus-8x7b-v2-0 2025-06-06 21:09:27 87.0 GB /var/home/azureuser/.cache/instructlab/models/prometheus-8x7b-v2-0

      The Warning received during ILAB MODEL SERVE

        WARNING 06-09 14:02:47 [api_server.py:936] Using supplied chat template:

      Unknown macro: {% set eos_token = "<|end_of_text|>" %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {% set bos_token = "<|end_of_text|>" %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if messages[0]['role'] == 'system' %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = messages[0]['content'] %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set loop_messages = messages[1}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- else %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = "Knowledge Cutoff Date}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if tools and documents %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = system_message + " You are a helpful AI assistant with access to the following tools. When a tool is required to answer the user's query, respond with <tool_call> followed by a JSON list of tools used. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request. WARNING 06-09 14}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- elif tools %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = system_message + " You are a helpful AI assistant with access to the following tools. When a tool is required to answer the user's query, respond with <tool_call> followed by a JSON list of tools used. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request." %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- elif documents %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = system_message + " Write the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data." %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = system_message + " Your primary role is to serve as a chat assistant." %}


      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- endif %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if 'citations' in controls and documents %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = system_message + ' WARNING 06-09 14}

      WARNING 06-09 14:02:47 [api_server.py:936]

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if 'hallucinations' in controls and documents %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set system_message = system_message + ' WARNING 06-09 14}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- endif %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- set loop_messages = messages %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|start_of_role|>system<|end_of_role|>' + system_message + '<|end_of_text|>
      WARNING 06-09 14:02:47 [api_server.py:936] ' }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if tools %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|start_of_role|>tools<|end_of_role|>' }}
      WARNING 06-09 14:02:47 [api_server.py:936] {{- tools | tojson(indent=4) }}
      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|end_of_text|>
      WARNING 06-09 14:02:47 [api_server.py:936] ' }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- endif %}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if documents %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|start_of_role|>documents<|end_of_role|>' }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- for document in documents %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- 'Document ' + loop.index0 | string + '
      WARNING 06-09 14:02:47 [api_server.py:936] ' }}
      WARNING 06-09 14:02:47 [api_server.py:936] {{- document['text'] }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if not loop.last %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '
      WARNING 06-09 14:02:47 [api_server.py:936]
      WARNING 06-09 14:02:47 [api_server.py:936] '}}
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- endif%}

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- endfor %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|end_of_text|>
      WARNING 06-09 14:02:47 [api_server.py:936] ' }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- for message in loop_messages %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|start_of_role|>' + message['role'] + '<|end_of_role|>' + message['content'] + '<|end_of_text|>
      WARNING 06-09 14:02:47 [api_server.py:936] ' }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if loop.last and add_generation_prompt %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|start_of_role|>assistant' }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- if controls %}

      WARNING 06-09 14:02:47 [api_server.py:936] - ' ' + controls | tojson()
      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- endif %}

      WARNING 06-09 14:02:47 [api_server.py:936] {{- '<|end_of_role|>' }}
      WARNING 06-09 14:02:47 [api_server.py:936]

      WARNING 06-09 14:02:47 [api_server.py:936]

      Unknown macro: {%- endfor %}

      WARNING 06-09 14:02:47 [api_server.py:936] It is different from official chat template '/var/home/azureuser/.cache/instructlab/models/granite-3.1-8b-lab-v2.1'. This discrepancy may lead to performance degradation.
      INFO 06-09 14:02:47 [api_server.py:1081] Starting vLLM API server on http://127.0.0.1:8000

       

              Unassigned Unassigned
              rh-ee-vshaw Vikash Shaw
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: