INFO 2025-11-13 13:05:07,822 uvicorn.access:473 uncategorized: [LOCAL-IP] - "GET /v1/models HTTP/1.1" 200 INFO 2025-11-13 13:05:07,833 uvicorn.access:473 uncategorized: [LOCAL-IP] - "GET /v1/shields HTTP/1.1" 200 INFO 2025-11-13 13:05:07,836 uvicorn.access:473 uncategorized: [LOCAL-IP] - "GET /v1/shields HTTP/1.1" 200 DEBUG 2025-11-13 13:05:07,842 llama_stack.core.server.server:214 server: Incoming raw request body for POST /v1/agents: { 'agent_config': { 'model': 'vllm/gpt-4.1', 'instructions': 'You are a helpful assistant', 'toolgroups': [], 'client_tools': [], 'enable_session_persistence': True, 'input_shields': ['lightspeed_question_validity-shield'], 'output_shields': [] } } INFO 2025-11-13 13:05:07,851 uvicorn.access:473 uncategorized: [LOCAL-IP] - "POST /v1/agents HTTP/1.1" 200 DEBUG 2025-11-13 13:05:07,855 llama_stack.core.server.server:214 server: Incoming raw request body for POST /v1/agents/[AGENT-ID]/session: {'session_name': '[SESSION-NAME-ID]'} INFO 2025-11-13 13:05:07,864 uvicorn.access:473 uncategorized: [LOCAL-IP] - "POST /v1/agents/[AGENT-ID]/session HTTP/1.1" 200 INFO 2025-11-13 13:05:07,869 uvicorn.access:473 uncategorized: [LOCAL-IP] - "GET /v1/vector-dbs HTTP/1.1" 200 DEBUG 2025-11-13 13:05:07,873 llama_stack.core.server.server:214 server: Incoming raw request body for POST /v1/agents/[AGENT-ID]/session/[SESSION-ID]/turn: { 'messages': [ { 'content': "using the backstage action tool, show me all templates of type 'service'", 'role': 'user' } ], 'documents': [], 'stream': True, 'toolgroups': [ { 'name': 'builtin::rag/knowledge_search', 'args': {'vector_db_ids': ['rhdh-product-docs-1_7']} }, 'mcp::backstage' ] } INFO 2025-11-13 13:05:07,875 uvicorn.access:473 uncategorized: [LOCAL-IP] - "POST /v1/agents/[AGENT-ID]/session/[SESSION-ID]/turn HTTP/1.1" 200 DEBUG 2025-11-13 13:05:07,884 llama_stack.core.routers.safety:55 core: SafetyRouter.run_shield: lightspeed_question_validity-shield DEBUG 2025-11-13 13:05:07,886 llama_stack.core.routers.inference:202 inference: InferenceRouter.chat_completion: model_id='vllm/gpt-4o', stream=False, messages=[UserMessage(role='user', content="Instructions:\n\nYou area question classification tool. You are an expert in the following categories:\n- Backstage\n- Red Hat Developer Hub (RHDH)\n- Kubernetes\n- Openshift\n- CI/CD\n- GitOps\n- Pipelines\n- Developer Portals\n- Deployments\n- Software Catalogs\n- Software Templates\n- Tech Docs\n\nYour job is to determine if a user's question is related to the categories you are an expert in. If the question is related to those categories, \\\nor any features that may be related to those categories, you will answer with ALLOWED.\n\nIf a question is not related to your expert categories, answer with REJECTED.\n\nYou do not need to explain your answer.\n\nBelow are some example questions:\nExample Question:\nWhy is the sky blue?\nExample Response:\nREJECTED\n\nExample Question:\nCan you help configure my cluster to automatically scale?\nExample Response:\nALLOWED\n\nExample Question:\nHow do I create import an existing software template in Backstage?\nExample Response:\nALLOWED\n\nExample Question:\nHow do I accomplish a task in RHDH?\nExample Response:\nALLOWED\n\nExample Question:\nHow do I explore a component in RHDH catalog?\nExample Response:\nALLOWED\n\nExample Question:\nHow can I integrate GitOps into my pipeline?\nExample Response:\nALLOWED\n\nQuestion:\nusing the backstage action tool, show me all templates of type 'service'\nResponse:", context=None)], tools=None, tool_config=None, response_format=None INFO 2025-11-13 13:05:09,895 httpx:1740 uncategorized: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" DEBUG 2025-11-13 13:05:09,924 llama_stack.core.routers.inference:202 inference: InferenceRouter.chat_completion: model_id='vllm/gpt-4.1', stream=True, messages=[SystemMessage(role='system', content='You are a helpful assistant'), UserMessage(role='user', content="using the backstage action tool, show me all templates of type 'service'", context=None)], tools=[ToolDefinition(tool_name='knowledge_search', description='Search for information in a database.', parameters={'query': ToolParamDefinition(param_type='string', description='The query to search for. Can be a natural language sentence or keywords.', required=True, default=None)}), ToolDefinition(tool_name='fetch-catalog-entities', description='Search and retrieve catalog entities from the Backstage server.\n\nList all Backstage entities such as Components, Systems, Resources, APIs, Locations, Users, and Groups. \nBy default, results are returned in JSON array format, where each entry in the JSON array is an entity with the following fields: \'name\', \'description\',\'type\', \'owner\', \'tags\', \'dependsOn\' and \'kind\'.\nSetting \'verbose\' to true will return the full Backstage entity objects, but should only be used if the reduced output is not sufficient, as this will significantly impact context usage (especially on smaller models).\nNote: \'type\' can only be filtered on if a specified entity \'kind\' is also specified.\n\nExample invocations and the output from those invocations:\n # Find all Resources of type storage\n fetch-catalog-entities kind:Resource type:storage\n Output: {\n "entities": [\n {\n "name": "ibm-granite-s3-bucket",\n "kind": "Resource",\n "type": "storage",\n "tags": [\n "genai",\n "ibm",\n "llm",\n "granite",\n "conversational",\n "task-text-generation"\n ]\n }\n ]\n\n\n', parameters={'kind': ToolParamDefinition(param_type='string', description='Filter entities by kind (e.g., Component, API, System)', required=True, default=None), 'type': ToolParamDefinition(param_type='string', description='Filter entities by type (e.g., ai-model, library, website).', required=True, default=None), 'name': ToolParamDefinition(param_type='string', description='Filter entities by name', required=True, default=None), 'owner': ToolParamDefinition(param_type='string', description='Filter entities by owner (e.g., team-platform, user:john.doe)', required=True, default=None), 'lifecycle': ToolParamDefinition(param_type='string', description='Filter entities by lifecycle (e.g., production, staging, development)', required=True, default=None), 'tags': ToolParamDefinition(param_type='string', description='Filter entities by tags as comma-separated values (e.g., "genai,ibm,llm,granite,conversational,task-text-generation")', required=True, default=None), 'verbose': ToolParamDefinition(param_type='boolean', description='If true, returns the full Backstage Entity object from the API rather than the shortened output.', required=True, default=None)}), ToolDefinition(tool_name='fetch-techdocs', description='Search and retrieve all TechDoc entities from the Backstage Server\n\n List all Backstage entities with techdocs. Results are returned in JSON array format, where each\n entry includes entity details and TechDocs metadata, like last update timestamp and build information.\n\n Example invocations and the output from those invocations:\n Output: {\n "entities": [\n {\n "name": "developer-model-service",\n "title": "Developer Model Service",\n "tags": [\n "genai",\n "ibm-granite"\n ],\n "description": "A description",\n "owner": "user:default/exampleuser",\n "lifecycle": "experimental",\n "namespace": "default",\n "kind": "Component",\n "techDocsUrl": "https://backstage.example.com/docs/default/component/developer-model-service",\n "metadataUrl": "https://backstage.example.com/api/techdocs/default/component/developer-model-service",\n "metadata": {\n "lastUpdated": "2024-01-15T10:30:00Z",\n "buildTimestamp": 1705313400,\n "siteName": "Developer Model Service Docs",\n "siteDescription": "Documentation for the developer model service"\n }\n }\n ]\n }\n }\n', parameters={'entityType': ToolParamDefinition(param_type='string', description='Filter by entity type (e.g., Component, API, System)', required=True, default=None), 'namespace': ToolParamDefinition(param_type='string', description='Filter by namespace', required=True, default=None), 'owner': ToolParamDefinition(param_type='string', description='Filter by owner (e.g., team-platform, user:john.doe)', required=True, default=None), 'lifecycle': ToolParamDefinition(param_type='string', description='Filter by lifecycle (e.g., production, staging, development)', required=True, default=None), 'tags': ToolParamDefinition(param_type='string', description='Filter by tags as comma-separated values (e.g., "genai,frontend,api")', required=True, default=None)}), ToolDefinition(tool_name='analyze-techdocs-coverage', description='Analyze documentation coverage across Backstage entities to understand what percentage of entities have TechDocs available.\n\n It calculates the percentage of entities that have TechDocs configured, helping identify documentation gaps and improve overall documentation coverage.\n\n Example output:\n {\n "totalEntities": 150,\n "entitiesWithDocs": 95,\n "coveragePercentage": 63.3\n }\n\n Supports filtering by entity type, namespace, owner, lifecycle, and tags to analyze coverage for specific subsets of entities.', parameters={'entityType': ToolParamDefinition(param_type='string', description='Filter by entity type (e.g., Component, API, System)', required=True, default=None), 'namespace': ToolParamDefinition(param_type='string', description='Filter by namespace', required=True, default=None), 'owner': ToolParamDefinition(param_type='string', description='Filter by owner (e.g., team-platform, user:john.doe)', required=True, default=None), 'lifecycle': ToolParamDefinition(param_type='string', description='Filter by lifecycle (e.g., production, staging, development)', required=True, default=None), 'tags': ToolParamDefinition(param_type='string', description='Filter by tags as comma-separated values (e.g., "genai,frontend,api")', required=True, default=None)}), ToolDefinition(tool_name='retrieve-techdocs-content', description='Retrieve the actual TechDocs content for a specific entity and optional page.\n\n This tool allows AI clients to access documentation content for specific catalog entities.\n You can retrieve the main documentation page or specific pages within the entity\'s documentation.\n\n Example invocations and expected responses:\n Input: {\n "entityRef": "component:default/developer-model-service",\n "pagePath": "index.html"\n }\n\n Output: {\n "entityRef": "component:default/developer-model-service",\n "name": "developer-model-service",\n "title": "Developer Model Service",\n "kind": "component",\n "namespace": "default",\n "content": "Developer Model Service Documentation\n\nWelcome to the service...",\n "pageTitle": "Developer Model Service Documentation",\n "path": "index.html",\n "contentType": "text",\n "lastModified": "2024-01-15T10:30:00Z",\n "metadata": {\n "lastUpdated": "2024-01-15T10:30:00Z",\n "buildTimestamp": 1705313400,\n "siteName": "Developer Model Service Docs"\n }\n }\n\n Note: HTML files are automatically converted to plain text for better readability and AI processing.\n Supports retrieving specific pages by providing pagePath parameter (e.g., "api/endpoints.html", "guides/setup.md").', parameters={'entityRef': ToolParamDefinition(param_type='string', description='Entity reference in format kind:namespace/name (e.g., component:default/my-service)', required=True, default=None), 'pagePath': ToolParamDefinition(param_type='string', description='Optional path to specific page within the documentation (defaults to index.html)', required=True, default=None)})], tool_config=ToolConfig(tool_choice=, tool_prompt_format=None, system_message_behavior=), response_format=None INFO 2025-11-13 13:05:10,656 httpx:1740 uncategorized: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO 2025-11-13 13:05:11,033 llama_stack.providers.inline.agents.meta_reference.agent_instance:892 agents: executing tool call: fetch-catalog-entities with args: {'kind': 'Template', 'type': 'service', 'name': '', 'owner': '', 'lifecycle': '', 'tags': '', 'verbose': False} DEBUG 2025-11-13 13:05:11,034 llama_stack.core.routers.tool_runtime:80 core: ToolRuntimeRouter.invoke_tool: fetch-catalog-entities INFO 2025-11-13 13:05:11,061 httpx:1740 uncategorized: HTTP Request: POST https://[REDACTED-HOSTNAME]/api/mcp-actions/v1 "HTTP/1.1 200 OK" INFO 2025-11-13 13:05:11,062 mcp.client.streamable_http:146 uncategorized: Negotiated protocol version: 2025-06-18 INFO 2025-11-13 13:05:11,078 httpx:1740 uncategorized: HTTP Request: POST https://[REDACTED-HOSTNAME]/api/mcp-actions/v1 "HTTP/1.1 202 Accepted" INFO 2025-11-13 13:05:11,129 httpx:1740 uncategorized: HTTP Request: POST https://[REDACTED-HOSTNAME]/api/mcp-actions/v1 "HTTP/1.1 200 OK" INFO 2025-11-13 13:05:11,159 httpx:1740 uncategorized: HTTP Request: POST https://[REDACTED-HOSTNAME]/api/mcp-actions/v1 "HTTP/1.1 200 OK" DEBUG 2025-11-13 13:05:11,163 llama_stack.providers.inline.agents.meta_reference.agent_instance:902 agents: tool call fetch-catalog-entities completed with result: content=[TextContentItem(type='text', text='```json\n{\n "entities": [\n {\n "name": "python-hello-world",\n "kind": "Template",\n "tags": "python,docker,github-actions,quay",\n "description": "Scaffolds a new Python web application with Flask, Docker and GitHub Actions for building and pushing to Quay.io",\n "type": "service",\n "owner": "platform-team",\n "dependsOn": ""\n }\n ]\n}\n```')] error_message=None error_code=0 metadata=None DEBUG 2025-11-13 13:05:11,167 llama_stack.core.routers.inference:202 inference: InferenceRouter.chat_completion: model_id='vllm/gpt-4.1', stream=True, messages=[SystemMessage(role='system', content='You are a helpful assistant'), UserMessage(role='user', content="using the backstage action tool, show me all templates of type 'service'", context=None), CompletionMessage(role='assistant', content='', stop_reason=, tool_calls=[ToolCall(call_id='call_4T0gF2o6elZV666HkXc3mNV6', tool_name='fetch-catalog-entities', arguments={'kind': 'Template', 'type': 'service', 'name': '', 'owner': '', 'lifecycle': '', 'tags': '', 'verbose': False}, arguments_json='{"kind":"Template","type":"service","name":"","owner":"","lifecycle":"","tags":"","verbose":false}')]), ToolResponseMessage(role='tool', call_id='call_4T0gF2o6elZV666HkXc3mNV6', content=[TextContentItem(type='text', text='```json\n{\n "entities": [\n {\n "name": "python-hello-world",\n "kind": "Template",\n "tags": "python,docker,github-actions,quay",\n "description": "Scaffolds a new Python web application with Flask, Docker and GitHub Actions for building and pushing to Quay.io",\n "type": "service",\n "owner": "platform-team",\n "dependsOn": ""\n }\n ]\n}\n```')])], tools=[ToolDefinition(tool_name='knowledge_search', description='Search for information in a database.', parameters={'query': ToolParamDefinition(param_type='string', description='The query to search for. Can be a natural language sentence or keywords.', required=True, default=None)}), ToolDefinition(tool_name='fetch-catalog-entities', description='Search and retrieve catalog entities from the Backstage server.\n\nList all Backstage entities such as Components, Systems, Resources, APIs, Locations, Users, and Groups. \nBy default, results are returned in JSON array format, where each entry in the JSON array is an entity with the following fields: \'name\', \'description\',\'type\', \'owner\', \'tags\', \'dependsOn\' and \'kind\'.\nSetting \'verbose\' to true will return the full Backstage entity objects, but should only be used if the reduced output is not sufficient, as this will significantly impact context usage (especially on smaller models).\nNote: \'type\' can only be filtered on if a specified entity \'kind\' is also specified.\n\nExample invocations and the output from those invocations:\n # Find all Resources of type storage\n fetch-catalog-entities kind:Resource type:storage\n Output: {\n "entities": [\n {\n "name": "ibm-granite-s3-bucket",\n "kind": "Resource",\n "type": "storage",\n "tags": [\n "genai",\n "ibm",\n "llm",\n "granite",\n "conversational",\n "task-text-generation"\n ]\n }\n ]\n\n\n', parameters={'kind': ToolParamDefinition(param_type='string', description='Filter entities by kind (e.g., Component, API, System)', required=True, default=None), 'type': ToolParamDefinition(param_type='string', description='Filter entities by type (e.g., ai-model, library, website).', required=True, default=None), 'name': ToolParamDefinition(param_type='string', description='Filter entities by name', required=True, default=None), 'owner': ToolParamDefinition(param_type='string', description='Filter entities by owner (e.g., team-platform, user:john.doe)', required=True, default=None), 'lifecycle': ToolParamDefinition(param_type='string', description='Filter entities by lifecycle (e.g., production, staging, development)', required=True, default=None), 'tags': ToolParamDefinition(param_type='string', description='Filter entities by tags as comma-separated values (e.g., "genai,ibm,llm,granite,conversational,task-text-generation")', required=True, default=None), 'verbose': ToolParamDefinition(param_type='boolean', description='If true, returns the full Backstage Entity object from the API rather than the shortened output.', required=True, default=None)}), ToolDefinition(tool_name='fetch-techdocs', description='Search and retrieve all TechDoc entities from the Backstage Server\n\n List all Backstage entities with techdocs. Results are returned in JSON array format, where each\n entry includes entity details and TechDocs metadata, like last update timestamp and build information.\n\n Example invocations and the output from those invocations:\n Output: {\n "entities": [\n {\n "name": "developer-model-service",\n "title": "Developer Model Service",\n "tags": [\n "genai",\n "ibm-granite"\n ],\n "description": "A description",\n "owner": "user:default/exampleuser",\n "lifecycle": "experimental",\n "namespace": "default",\n "kind": "Component",\n "techDocsUrl": "https://backstage.example.com/docs/default/component/developer-model-service",\n "metadataUrl": "https://backstage.example.com/api/techdocs/default/component/developer-model-service",\n "metadata": {\n "lastUpdated": "2024-01-15T10:30:00Z",\n "buildTimestamp": 1705313400,\n "siteName": "Developer Model Service Docs",\n "siteDescription": "Documentation for the developer model service"\n }\n }\n ]\n }\n }\n', parameters={'entityType': ToolParamDefinition(param_type='string', description='Filter by entity type (e.g., Component, API, System)', required=True, default=None), 'namespace': ToolParamDefinition(param_type='string', description='Filter by namespace', required=True, default=None), 'owner': ToolParamDefinition(param_type='string', description='Filter by owner (e.g., team-platform, user:john.doe)', required=True, default=None), 'lifecycle': ToolParamDefinition(param_type='string', description='Filter by lifecycle (e.g., production, staging, development)', required=True, default=None), 'tags': ToolParamDefinition(param_type='string', description='Filter by tags as comma-separated values (e.g., "genai,frontend,api")', required=True, default=None)}), ToolDefinition(tool_name='analyze-techdocs-coverage', description='Analyze documentation coverage across Backstage entities to understand what percentage of entities have TechDocs available.\n\n It calculates the percentage of entities that have TechDocs configured, helping identify documentation gaps and improve overall documentation coverage.\n\n Example output:\n {\n "totalEntities": 150,\n "entitiesWithDocs": 95,\n "coveragePercentage": 63.3\n }\n\n Supports filtering by entity type, namespace, owner, lifecycle, and tags to analyze coverage for specific subsets of entities.', parameters={'entityType': ToolParamDefinition(param_type='string', description='Filter by entity type (e.g., Component, API, System)', required=True, default=None), 'namespace': ToolParamDefinition(param_type='string', description='Filter by namespace', required=True, default=None), 'owner': ToolParamDefinition(param_type='string', description='Filter by owner (e.g., team-platform, user:john.doe)', required=True, default=None), 'lifecycle': ToolParamDefinition(param_type='string', description='Filter by lifecycle (e.g., production, staging, development)', required=True, default=None), 'tags': ToolParamDefinition(param_type='string', description='Filter by tags as comma-separated values (e.g., "genai,frontend,api")', required=True, default=None)}), ToolDefinition(tool_name='retrieve-techdocs-content', description='Retrieve the actual TechDocs content for a specific entity and optional page.\n\n This tool allows AI clients to access documentation content for specific catalog entities.\n You can retrieve the main documentation page or specific pages within the entity\'s documentation.\n\n Example invocations and expected responses:\n Input: {\n "entityRef": "component:default/developer-model-service",\n "pagePath": "index.html"\n }\n\n Output: {\n "entityRef": "component:default/developer-model-service",\n "name": "developer-model-service",\n "title": "Developer Model Service",\n "kind": "component",\n "namespace": "default",\n "content": "Developer Model Service Documentation\n\nWelcome to the service...",\n "pageTitle": "Developer Model Service Documentation",\n "path": "index.html",\n "contentType": "text",\n "lastModified": "2024-01-15T10:30:00Z",\n "metadata": {\n "lastUpdated": "2024-01-15T10:30:00Z",\n "buildTimestamp": 1705313400,\n "siteName": "Developer Model Service Docs"\n }\n }\n\n Note: HTML files are automatically converted to plain text for better readability and AI processing.\n Supports retrieving specific pages by providing pagePath parameter (e.g., "api/endpoints.html", "guides/setup.md").', parameters={'entityRef': ToolParamDefinition(param_type='string', description='Entity reference in format kind:namespace/name (e.g., component:default/my-service)', required=True, default=None), 'pagePath': ToolParamDefinition(param_type='string', description='Optional path to specific page within the documentation (defaults to index.html)', required=True, default=None)})], tool_config=ToolConfig(tool_choice=, tool_prompt_format=None, system_message_behavior=), response_format=None INFO 2025-11-13 13:05:11,571 httpx:1740 uncategorized: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 400 Bad Request" ERROR 2025-11-13 13:05:11,572 llama_stack.core.server.server:194 server: Error in sse_generator ╭──────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/core/server/server.py:187 in sse_generator │ │ │ │ 184 │ event_gen = None │ │ 185 │ try: │ │ 186 │ │ event_gen = await event_gen_coroutine │ │ ❱ 187 │ │ async for item in event_gen: │ │ 188 │ │ │ yield create_sse_event(item) │ │ 189 │ except asyncio.CancelledError: │ │ 190 │ │ logger.info("Generator cancelled") │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/providers/inline/agents/meta_reference/agents.py:181 in │ │ _create_agent_turn_streaming │ │ │ │ 178 │ │ request: AgentTurnCreateRequest, │ │ 179 │ ) -> AsyncGenerator: │ │ 180 │ │ agent = await self._get_agent_impl(request.agent_id) │ │ ❱ 181 │ │ async for event in agent.create_and_execute_turn(request): │ │ 182 │ │ │ yield event │ │ 183 │ │ │ 184 │ async def resume_agent_turn( │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/providers/inline/agents/meta_reference/agent_instance.py:191 in │ │ create_and_execute_turn │ │ │ │ 188 │ │ │ │ span.set_attribute("agent_name", self.agent_config.name) │ │ 189 │ │ │ │ 190 │ │ await self._initialize_tools(request.toolgroups) │ │ ❱ 191 │ │ async for chunk in self._run_turn(request, turn_id): │ │ 192 │ │ │ yield chunk │ │ 193 │ │ │ 194 │ async def resume_turn(self, request: AgentTurnResumeRequest) -> AsyncGenerator: │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/providers/inline/agents/meta_reference/agent_instance.py:276 in _run_turn │ │ │ │ 273 │ │ │ input_messages = request.messages │ │ 274 │ │ │ │ 275 │ │ output_message = None │ │ ❱ 276 │ │ async for chunk in self.run( │ │ 277 │ │ │ session_id=request.session_id, │ │ 278 │ │ │ turn_id=turn_id, │ │ 279 │ │ │ input_messages=messages, │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/providers/inline/agents/meta_reference/agent_instance.py:349 in run │ │ │ │ 346 │ │ │ │ else: │ │ 347 │ │ │ │ │ yield res │ │ 348 │ │ │ │ ❱ 349 │ │ async for res in self._run( │ │ 350 │ │ │ session_id, │ │ 351 │ │ │ turn_id, │ │ 352 │ │ │ input_messages, │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/providers/inline/agents/meta_reference/agent_instance.py:513 in _run │ │ │ │ 510 │ │ │ async with tracing.span("inference") as span: │ │ 511 │ │ │ │ if self.agent_config.name: │ │ 512 │ │ │ │ │ span.set_attribute("agent_name", self.agent_config.name) │ │ ❱ 513 │ │ │ │ async for chunk in await self.inference_api.chat_completion( │ │ 514 │ │ │ │ │ self.agent_config.model, │ │ 515 │ │ │ │ │ input_messages, │ │ 516 │ │ │ │ │ tools=self.tool_defs, │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/providers/utils/telemetry/trace_protocol.py:87 in async_gen_wrapper │ │ │ │ 84 │ │ │ with tracing.span(f"{class_name}.{method_name}", span_attributes) as span: │ │ 85 │ │ │ │ try: │ │ 86 │ │ │ │ │ count = 0 │ │ ❱ 87 │ │ │ │ │ async for item in method(self, *args, **kwargs): │ │ 88 │ │ │ │ │ │ yield item │ │ 89 │ │ │ │ │ │ count += 1 │ │ 90 │ │ │ │ finally: │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/core/routers/inference.py:626 in stream_tokens_and_compute_metrics │ │ │ │ 623 │ │ tool_prompt_format: ToolPromptFormat | None = None, │ │ 624 │ ) -> AsyncGenerator[ChatCompletionResponseStreamChunk, None] | │ │ AsyncGenerator[CompletionResponseStreamChunk, None]: │ │ 625 │ │ completion_text = "" │ │ ❱ 626 │ │ async for chunk in response: │ │ 627 │ │ │ complete = False │ │ 628 │ │ │ if hasattr(chunk, "event"): # only ChatCompletions have .event │ │ 629 │ │ │ │ if chunk.event.event_type == ChatCompletionResponseEventType.progress: │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/llama_stack/providers/remote/inference/vllm/vllm.py:457 in _stream_chat_completion │ │ │ │ 454 │ ) -> AsyncGenerator[ChatCompletionResponseStreamChunk, None]: │ │ 455 │ │ params = await self._get_params(request) │ │ 456 │ │ │ │ ❱ 457 │ │ stream = await client.chat.completions.create(**params) │ │ 458 │ │ if request.tools: │ │ 459 │ │ │ res = _process_vllm_chat_completion_stream_response(stream) │ │ 460 │ │ else: │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/openai/resources/chat/completions/completions.py:2589 in create │ │ │ │ 2586 │ │ timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN, │ │ 2587 │ ) -> ChatCompletion | AsyncStream[ChatCompletionChunk]: │ │ 2588 │ │ validate_response_format(response_format) │ │ ❱ 2589 │ │ return await self._post( │ │ 2590 │ │ │ "/chat/completions", │ │ 2591 │ │ │ body=await async_maybe_transform( │ │ 2592 │ │ │ │ { │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/openai/_base_client.py:1794 in post │ │ │ │ 1791 │ │ opts = FinalRequestOptions.construct( │ │ 1792 │ │ │ method="post", url=path, json_data=body, files=await │ │ async_to_httpx_files(files), **options │ │ 1793 │ │ ) │ │ ❱ 1794 │ │ return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls) │ │ 1795 │ │ │ 1796 │ async def patch( │ │ 1797 │ │ self, │ │ │ │ /app-root/.venv/lib64/python3.12/site-packages/openai/_base_client.py:1594 in request │ │ │ │ 1591 │ │ │ │ │ await err.response.aread() │ │ 1592 │ │ │ │ │ │ 1593 │ │ │ │ log.debug("Re-raising status error") │ │ ❱ 1594 │ │ │ │ raise self._make_status_error_from_response(err.response) from None │ │ 1595 │ │ │ │ │ 1596 │ │ │ break │ │ 1597 │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ BadRequestError: Error code: 400 - {'error': {'message': "Missing parameter 'tool_call_id': messages with role 'tool' must have a 'tool_call_id'.", 'type': 'invalid_request_error', 'param': 'messages.[3].tool_call_id', 'code': None}}