[Bug]: AgentWorkflow doesn't call functions when using Ollama and llama3.2

[telcott]

Bug Description

A very simple test case demonstrates successful functional calling with OpenAI, but unsuccessful with Ollama and llama3.2.

`from llama_index.core.workflow import Context
from llama_index.core.agent.workflow import AgentWorkflow
from llama_index.llms.ollama import Ollama
from llama_index.llms.openai import OpenAI
import logging
import logging.config

logger = logging.getLogger(name)
logging.basicConfig(level=logging.INFO)

async def search_web(query: str) -> str:
"""Useful for using the web to answer questions."""
print("Search_web called with query:", query)

return str("""

• Temperature: 0.1°C (32.2°F)
• Condition: Partly cloudy
• Wind: 13.2 mph (21.2 kph) from the SSE
• Humidity: 80%
• Feels Like: -5.3°C (22.5°F)
• Visibility: 24 km (14 miles)
  """)

async def tell_joke(query: str) -> str:
"""Useful for telling a joke."""
print("tell_joke called with query:", query)

return str("""

Why did the chicken cross the road?
To get to the other side!""")

llm1 = OpenAI(model="gpt-4o-mini",
api_key="sk-XXXXXXXXXXX",
)

llm2 = Ollama(
model="llama3.2",
is_function_calling_model=True,
json_mode=True,
temperature=0.2
)

workflow = AgentWorkflow.from_tools_or_functions(
[search_web, tell_joke],
llm=llm1,
system_prompt="You are a helpful assistant that can search the web for information.")

workflow2 = AgentWorkflow.from_tools_or_functions(
[search_web, tell_joke],
llm=llm2,
system_prompt="You are a helpful assistant that can search the web for information. When using a tool, always state the data source and use ONLY the tool's response. Do not assume an answer.")

ctx1 = Context(workflow)
ctx2 = Context(workflow2)

async def run_workflow(user_msg: str) -> str:
response = await workflow.run(user_msg=user_msg, ctx=ctx1)
print(f"Response: {str(response)}")
print("Final response:", repr(response))

async def run_workflow2(user_msg: str) -> str:
response = await workflow2.run(user_msg=user_msg, ctx=ctx2)
print(f"Response: {str(response)}")
print("Final response:", repr(response))

llm=None

if name == "main":
import asyncio

asyncio.run(run_workflow("1- What is the weather in London?"))

asyncio.run(run_workflow2("2- What is the weather in London?"))

`

Version

0.12.15

Steps to Reproduce

Run the attached code

Relevant Logs/Tracbacks

pyhttpdbg log

OpenAI

Request
{"messages":[{"role":"system","content":"You are a helpful assistant that can search the web for information."},{"role":"user","content":"1- What is the weather in London?"}],"model":"gpt-4o-mini","stream":true,"temperature":0.1,"tool_choice":"auto","tools":[{"type":"function","function":{"name":"search_web","description":"search_web(query: str) -> str\nUseful for using the web to answer questions.","parameters":{"properties":{"query":{"title":"Query","type":"string"}},"required":["query"],"type":"object","additionalProperties":false},"strict":false}},{"type":"function","function":{"name":"tell_joke","description":"tell_joke(query: str) -> str\nUseful for telling a joke.","parameters":{"properties":{"query":{"title":"Query","type":"string"}},"required":["query"],"type":"object","additionalProperties":false},"strict":false}}]}

Reponse

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_rzPolF5BObgOH7kWYjcdAKd0","type":"function","function":{"name":"search_web","arguments":""}}],"refusal":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"query"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"current"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":" weather"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":" in"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":" London"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-AxMsLkr5aQmjKrkXjLAGK8Po7VZWL","object":"chat.completion.chunk","created":1738712649,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_72ed7ab54c","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"tool_calls"}]}

data: [DONE]

Ollama

Request

{"model":"llama3.2","stream":true,"options":{"temperature":0.2,"num_ctx":3900},"format":"json","messages":[{"role":"system","content":"You are a helpful assistant that can search the web for information. When using a tool, always state the data source and use ONLY the tool's response. Do not assume an answer."},{"role":"user","content":"2- What is the weather in London?"}],"tools":[{"type":"function","function":{"name":"search_web","description":"search_web(query: str) -> str\nUseful for using the web to answer questions.","parameters":{"type":"object","required":["query"],"properties":{"query":{"type":"string"}}}}},{"type":"function","function":{"name":"tell_joke","description":"tell_joke(query: str) -> str\nUseful for telling a joke.","parameters":{"type":"object","required":["query"],"properties":{"query":{"type":"string"}}}}}]}

Response

{"model":"llama3.2","created_at":"2025-02-04T23:44:12.942294Z","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"search_web","arguments":{"query":"London weather"}}}]},"done":false}
{"model":"llama3.2","created_at":"2025-02-04T23:44:12.949621Z","message":{"role":"assistant","content":""},"done_reason":"stop","done":true,"total_duration":1155114250,"load_duration":582023333,"prompt_eval_count":272,"prompt_eval_duration":227000000,"eval_count":18,"eval_duration":127000000}

[dosubot]

@telcott Hi there! I'm here to help you with any issues or questions you have. Let me know how I can assist you today.

The issue you're experiencing with the AgentWorkflow not calling functions when using the Ollama model (llama3.2) could be related to known limitations with the function calling capabilities of the Ollama model in the LlamaIndex framework. Specifically, there are issues with function calling when using certain features, such as SchemaLLMPathExtractor, which may not work correctly with local models like Ollama due to API communication issues or function calling limitations ^[1].

In your case, the Ollama model seems to be generating the correct tool call in the response, but it might not be executing it as expected. This could be due to how the function calling mechanism is implemented for the Ollama model compared to OpenAI's models. You might want to verify if the function calling setup for Ollama is correctly configured and if there are any additional settings or parameters that need to be adjusted for it to work as intended.

Additionally, ensure that the is_function_calling_model parameter is correctly set to True for the Ollama model, as this indicates that the model supports function calling capabilities ^[2]. If the issue persists, consider checking for any updates or patches in the LlamaIndex framework that might address this specific problem.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[logan-markewich]

From the logs it pretty clearly calls the search_web tool with ollama

What's the issue?

[logan-markewich]

Looks like ollama just streams tool calls in less pieces compared to openai

[telcott]

Thanks for looking at this. Both OpenAI and Ollama make the right tool decision, but, when using OpenAI it actually calls the search_web method, where as when running with Ollama, it doesn’t call search_web.

[telcott]

Here's the test python file, requirements.txt, and a debug.log

files 2025-02-05.zip

[logan-markewich]

Great, found the bug. Will be fixed shortly

[telcott]

Awesome, Logan. Massively appreciated!

[logan-markewich]

pip install -U llama-index-llms-ollama