[Bug]: langchain qwen2.5 function calling error

[Valdanitooooo]

Your current environment

The output of `python collect_env.py`

vllm v0.6.3

Model Input Dumps

Test code:

test_llm = ChatOpenAI(
    openai_api_key="xxx",
    openai_api_base="http://xxxxx:6000/v1",
    model_name="Qwen2.5-7B-Instruct",
    temperature=0.01,
    max_tokens=1024,
    streaming=True,
    max_retries=0,
    request_timeout=60,
    callbacks=[],
    frequency_penalty=1.05,
    model_kwargs={"stop": ["<|endoftext|>", "<|im_end|>"], "extra_body": {"skip_special_tokens": False}},
)

agent = create_react_agent(
    model=test_llm,
    tools=[get_current_time, get_current_hour, get_current_date, get_current_week, get_current_month,
           get_current_year],
    debug=True,
)

async for chunk in agent.astream(
        input={"messages": [HumanMessage(content="现在是几月份")]},
        config=RunnableConfig(
            configurable={"thread_id": 1},
            recursion_limit=15,
        ),
):
    print(chunk)

Output:

{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'chatcmpl-tool-62111907b5634a5ca584761b1528a2be', 'function': {'arguments': None, 'name': 'get_current_month'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls', 'model_name': 'Qwen2.5-7B-Instruct'}, id='run-e81f28b5-51f6-4114-8fa6-90b4e9fee7c6-0', invalid_tool_calls=[{'name': 'get_current_month', 'args': None, 'id': 'chatcmpl-tool-62111907b5634a5ca584761b1528a2be', 'error': None, 'type': 'invalid_tool_call'}])]}}

🐛 Describe the bug

Error:

ERROR 10-17 01:19:09 hermes_tool_parser.py:337] Error trying to handle streaming tool call: 'NoneType' object has no attribute 'get'
ERROR 10-17 01:19:09 hermes_tool_parser.py:337] Error trying to handle streaming tool call: 'NoneType' object has no attribute 'get'
ERROR 10-17 01:19:09 hermes_tool_parser.py:337] Error trying to handle streaming tool call: cannot access local variable 'tool_call_portion' where it is not associated with a value

logs:

INFO 10-17 01:19:09 logger.py:37] Received request chat-b369c5f37db34326a874f158dce0b6e1: prompt: '<|im_start|>system\nYou are a helpful assistant.\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{"type": "function", "function": {"name": "get_current_time", "description": "获取当前时间", "parameters": {"properties": {}, "type": "object"}}}\n{"type": "function", "function": {"name": "get_current_hour", "description": "获取当前的小时数（24小时制）", "parameters": {"properties": {}, "type": "object"}}}\n{"type": "function", "function": {"name": "get_current_date", "description": "获取当前日期", "parameters": {"properties": {}, "type": "object"}}}\n{"type": "function", "function": {"name": "get_current_week", "description": "获取当前周数", "parameters": {"properties": {}, "type": "object"}}}\n{"type": "function", "function": {"name": "get_current_month", "description": "获取当前月份", "parameters": {"properties": {}, "type": "object"}}}\n{"type": "function", "function": {"name": "get_current_year", "description": "获取当前年份", "parameters": {"properties": {}, "type": "object"}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": <function-name>, "arguments": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\n现在是几月份<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=1.05, repetition_penalty=1.0, temperature=0.01, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=['<|endoftext|>', '<|im_end|>'], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=False, spaces_between_special_tokens=True, truncate_prompt_tokens=None), guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), prompt_token_ids: [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 382, 2, 13852, 271, 2610, 1231, 1618, 825, 476, 803, 5746, 311, 7789, 448, 279, 1196, 3239, 382, 2610, 525, 3897, 448, 729, 32628, 2878, 366, 15918, 1472, 15918, 29, 11874, 9492, 510, 27, 15918, 397, 4913, 1313, 788, 330, 1688, 497, 330, 1688, 788, 5212, 606, 788, 330, 455, 11080, 3009, 497, 330, 4684, 788, 330, 45912, 67949, 20450, 497, 330, 13786, 788, 5212, 13193, 788, 16452, 330, 1313, 788, 330, 1700, 30975, 532, 4913, 1313, 788, 330, 1688, 497, 330, 1688, 788, 5212, 606, 788, 330, 455, 11080, 32292, 497, 330, 4684, 788, 330, 45912, 67949, 9370, 92536, 8863, 9909, 17, 19, 92536, 43316, 7552, 497, 330, 13786, 788, 5212, 13193, 788, 16452, 330, 1313, 788, 330, 1700, 30975, 532, 4913, 1313, 788, 330, 1688, 497, 330, 1688, 788, 5212, 606, 788, 330, 455, 11080, 4164, 497, 330, 4684, 788, 330, 45912, 67949, 45785, 497, 330, 13786, 788, 5212, 13193, 788, 16452, 330, 1313, 788, 330, 1700, 30975, 532, 4913, 1313, 788, 330, 1688, 497, 330, 1688, 788, 5212, 606, 788, 330, 455, 11080, 31277, 497, 330, 4684, 788, 330, 45912, 67949, 40542, 8863, 497, 330, 13786, 788, 5212, 13193, 788, 16452, 330, 1313, 788, 330, 1700, 30975, 532, 4913, 1313, 788, 330, 1688, 497, 330, 1688, 788, 5212, 606, 788, 330, 455, 11080, 18933, 497, 330, 4684, 788, 330, 45912, 67949, 104081, 497, 330, 13786, 788, 5212, 13193, 788, 16452, 330, 1313, 788, 330, 1700, 30975, 532, 4913, 1313, 788, 330, 1688, 497, 330, 1688, 788, 5212, 606, 788, 330, 455, 11080, 14645, 497, 330, 4684, 788, 330, 45912, 67949, 7948, 69442, 497, 330, 13786, 788, 5212, 13193, 788, 16452, 330, 1313, 788, 330, 1700, 30975, 532, 522, 15918, 1339, 2461, 1817, 729, 1618, 11, 470, 264, 2951, 1633, 448, 729, 829, 323, 5977, 2878, 220, 151657, 151658, 11874, 9492, 510, 151657, 198, 4913, 606, 788, 366, 1688, 11494, 8066, 330, 16370, 788, 366, 2116, 56080, 40432, 31296, 151658, 151645, 198, 151644, 872, 198, 99601, 20412, 99195, 104081, 151645, 198, 151644, 77091, 198], lora_request: None, prompt_adapter_request: None.
INFO:     10.20.1.3:64491 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 10-17 01:19:09 engine.py:292] Added request chat-b369c5f37db34326a874f158dce0b6e1.
INFO 10-17 01:19:09 metrics.py:345] Avg prompt throughput: 49.7 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
ERROR 10-17 01:19:09 hermes_tool_parser.py:337] Error trying to handle streaming tool call: 'NoneType' object has no attribute 'get'
ERROR 10-17 01:19:09 hermes_tool_parser.py:337] Error trying to handle streaming tool call: 'NoneType' object has no attribute 'get'
ERROR 10-17 01:19:09 hermes_tool_parser.py:337] Error trying to handle streaming tool call: cannot access local variable 'tool_call_portion' where it is not associated with a value

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[frei-x]

me too

[Jimmy-L99]

@Valdanitooooo
能请教一下你运行命令的参数吗，我看文档的vllm serve启动服务，如果要用工具貌似需要参数--enable-auto-tool-choice、--tool-call-parser和--tool-call-parser，这里要怎么填？

[Valdanitooooo]

--enable-auto-tool-choice、--tool-call-parser和--tool-call-parser，这里要怎么填？

@2500035435 我就是按官方教程填的

--enable-auto-tool-choice --tool-call-parser hermes

[Jimmy-L99]

--enable-auto-tool-choice、--tool-call-parser和--tool-call-parser，这里要怎么填？

@2500035435 我就是按官方教程填的

--enable-auto-tool-choice --tool-call-parser hermes

我用了hermes，看您上面还能出来response，我这边直接报错了: (

llm = ChatOpenAI(
    api_key="api_key", 
    temperature=0.1, 
    top_p=0.7,
    max_tokens=8192,
    model="glm4-9b-chat-lora1", 
    base_url="http://172.16.21.155:8003/v1/"
)

agent = create_react_agent(llm, tools=[weather], debug=True)

async for chunk in agent.astream(
        input={"messages": [HumanMessage(content="今天深圳天气怎么样？")]},
        config=RunnableConfig(
            configurable={"thread_id": 1},
            recursion_limit=15,
        ),
):
    print(chunk)

[-2:checkpoint] State at the end of step -2:
{'messages': []}
[0:tasks] Starting step 0 with 1 task:

• start -> {'messages': [HumanMessage(content='今天深圳天气怎么样？')]}
  [0:writes] Finished step 0 with writes to 1 channel:
• messages -> [HumanMessage(content='今天深圳天气怎么样？')]
  [-2:checkpoint] State at the end of step -2:
  {'messages': [HumanMessage(content='今天深圳天气怎么样？', id='c0623ede-904b-4750-bf62-c475dafcdd6f')]}
  [1:tasks] Starting step 1 with 1 task:
• agent -> {'is_last_step': False,
  'messages': [HumanMessage(content='今天深圳天气怎么样？', id='c0623ede-904b-4750-bf62-c475dafcdd6f')]}

InternalServerError: Error code: 500

[Valdanitooooo]

@2500035435 你用的 glm4-9b，咱们不是一个模型

[Jimmy-L99]

@2500035435 你用的 glm4-9b，咱们不是一个模型

是的，我以为vllm只要别的模型能通过langchain框架调用工具，我这也行😨

[Valdanitooooo]

@2500035435 我使用 vllm qwen2 & langchain 是没问题的，我一直在用，就是现在没法升级 qwen2.5

[frei-x]

0.6.2 是不是不会出现问题, 函数是空参数

[Valdanitooooo]

@frei-x 我换了这个模板可以了 https://github.com/chujiezheng/chat_templates/blob/main/chat_templates/qwen2.5-instruct.jinja

[frei-x]

@Valdanitooooo 感谢分享

[bulat15g]

thanks
I see chat templates