[Bug]: WorkflowRuntimeError: Error in step 'run_agent_step': when running non OpenAI LLM

[pazevedo-hyland]

Bug Description

Getting this error WorkflowRuntimeError: Error in step 'run_agent_step': 'toolUse' whenever I run an non OpenAI LLM such as Bedrock Converse on new Agent Workflows

Version

0.12.14

Steps to Reproduce

Use any tool and define as such:

from llama_index.llms.bedrock import Bedrock
from llama_index.llms.bedrock_converse import BedrockConverse

from llama_index.core.agent.workflow import FunctionAgent, ReActAgent
from llama_index.core.agent.workflow import AgentWorkflow

llm= BedrockConverse(
    model="anthropic.claude-3-haiku-20240307-v1:0",
    #context_size=300000
    
)

workflow = AgentWorkflow.from_tools_or_functions(
    [tavily_tool.to_tool_list()[0]],
    llm=llm,
    system_prompt=(
        "You are a helpful assistant that can search the web for information."
    )
)

# Run the agent
response = await workflow.run(
    user_msg="What is the latest news about AI?",
)

Relevant Logs/Tracbacks

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/cin-ml-agent/.venv/lib/python3.12/site-packages/llama_index/core/workflow/workflow.py:247, in Workflow._start.<locals>._task(name, queue, step, config)
    246 try:
--> 247     new_ev = await instrumented_step(**kwargs)
    248     break  # exit the retrying loop

File ~/cin-ml-agent/.venv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py:367, in Dispatcher.span.<locals>.async_wrapper(func, instance, args, kwargs)
    366 try:
--> 367     result = await func(*args, **kwargs)
    368 except BaseException as e:

File ~/cin-ml-agent/.venv/lib/python3.12/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:290, in AgentWorkflow.run_agent_step(self, ctx, ev)
    288 tools = await self.get_tools(ev.current_agent_name, ev.input[-1].content or "")
--> 290 agent_output = await agent.take_step(
    291     ctx,
    292     ev.input,
    293     tools,
    294     memory,
    295 )
    297 ctx.write_event_to_stream(agent_output)

File ~/cin-ml-agent/.venv/lib/python3.12/site-packages/llama_index/core/agent/workflow/function_agent.py:43, in FunctionAgent.take_step(self, ctx, llm_input, tools, memory)
     40 response = await self.llm.astream_chat_with_tools(  # type: ignore
     41     tools, chat_history=current_llm_input, allow_parallel_tool_calls=True
...
    260     )
    261     if delay is None:
    262         # We're done retrying

WorkflowRuntimeError: Error in step 'run_agent_step': 'toolUse'

[dosubot]

Hey @pazevedo-hyland! I'm here to help you with any bugs, questions, or contributions you have. I'll assist you while we wait for a human maintainer.

To resolve the WorkflowRuntimeError: Error in step 'run_agent_step': 'toolUse' when using a non-OpenAI LLM like Bedrock Converse in Agent Workflows with the llama_index package, you can follow these steps:

1. Update to llama-index-core v0.12.14 and use ReActAgent instead of FunctionAgent by modifying the agent class selection in your code:

   agent_cls = ReActAgent

2. Alternatively, use a modified FunctionAgent that utilizes achat_with_tools instead of stream_chat_with_tools. This approach requires changing tool responses from assistant to user messages to avoid validation errors with Bedrock models.

These solutions address the KeyError: 'toolUse' error when using BedrockConverse with AgentWorkflow.from_tools_or_functions() ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[Kobtul]

This is the same issue as mine:
#17616

[naseemshah]

Yes, I have same issue.

from llama_index.core.tools import QueryEngineTool, ToolMetadata

# Create Query Engines from the loaded indices
vector_query_engine = vector_index.as_query_engine()
summary_query_engine = summary_index.as_query_engine()

# Define VectorStoreTool for detailed review queries
vector_store_tool = QueryEngineTool(
    query_engine=vector_query_engine,
    metadata=ToolMetadata(
        name="ReviewsVectorStoreTool",
        description=(
            "Provides detailed information based on customer reviews. "
            "Use this tool to ask specific questions about the reviews data."
        ),
    ),# Set to True if you want to return results directly without agent processing
)

# Define SummaryTool for summarized review queries
summary_tool = QueryEngineTool(
    query_engine=summary_query_engine,
    metadata=ToolMetadata(
        name="ReviewsSummaryTool",
        description=(
            "Provides summaries and overviews of customer reviews. "
            "Use this tool to request high-level explanations or summaries of the reviews."
        ),
    )
)


tools = [vector_store_tool, summary_tool]

from llama_index.core.agent.workflow import (
    AgentWorkflow, ReActAgent, AgentStream
)


# Initialize AgentWorkflow with ReActAgent
agent_workflow = AgentWorkflow(
    agents=[
        ReActAgent(
            name="reviews_react_agent",
            description="Agent to handle review-related queries.",
            tools=tools,
            llm=vertex_gemini,          # Using Vertex as configured
            system_prompt="You are an assistant that helps answer questions based on customer reviews.",  # Optional: Customize as needed
        )
    ],
)

handler = await agent_workflow.run(user_msg="i head some reviwes mention about cancer issues")
async for event in handler.stream_events():
    if hasattr(event, "delta"):
        print(event.delta, end="", flush=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/workflow/workflow.py:247, in Workflow._start.<locals>._task(name, queue, step, config)
    [246](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/workflow/workflow.py:246) try:
--> [247](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/workflow/workflow.py:247)     new_ev = await instrumented_step(**kwargs)
    [248](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/workflow/workflow.py:248)     break  # exit the retrying loop

File ~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py:367, in Dispatcher.span.<locals>.async_wrapper(func, instance, args, kwargs)
    [366](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py:366) try:
--> [367](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py:367)     result = await func(*args, **kwargs)
    [368](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py:368) except BaseException as e:

File ~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:290, in AgentWorkflow.run_agent_step(self, ctx, ev)
    [288](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:288) tools = await self.get_tools(ev.current_agent_name, ev.input[-1].content or "")
--> [290](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:290) agent_output = await agent.take_step(
    [291](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:291)     ctx,
    [292](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:292)     ev.input,
    [293](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:293)     tools,
    [294](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:294)     memory,
    [295](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:295) )
    [297](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/multi_agent_workflow.py:297) ctx.write_event_to_stream(agent_output)

File ~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/react_agent.py:92, in ReActAgent.take_step(self, ctx, llm_input, tools, memory)
     [91](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/react_agent.py:91) # Initial LLM call
---> [92](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/agent/workflow/react_agent.py:92) response = await self.llm.astream_chat(input_chat)
...
    [260](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/workflow/workflow.py:260)     )
    [261](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/workflow/workflow.py:261)     if delay is None:
    [262](https://file+.vscode-resource.vscode-cdn.net/Users/naseemshah/coding/brandvoice-rag-test/notebooks/~/coding/brandvoice-rag-test/notebooks/.venv/lib/python3.10/site-packages/llama_index/core/workflow/workflow.py:262)         # We're done retrying

WorkflowRuntimeError: Error in step 'run_agent_step': Not Implemented
Output is truncated. Vie

[logan-markewich]

@naseemshah that is a different error. Something is not implemented in vertex (async probably? I'd have to check)

@pazevedo-hyland @Kobtul i would appreciate any help in making a fix to the bedrock converse class. Seems like how tools work has updated in the converse api

[naseemshah]

@logan-markewich Seems like Vertex AI has alot of compability issues (or SDK/API changes), will see if I can contribute a PR.

[logan-markewich]

@naseemshah that would be great if you could! ❤

[logan-markewich]

@pazevedo-hyland I've fixed this issue now in the above PR. When it merges, it will publish a new version of llama-index-llms-bedrock-converse