Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 8 additions & 13 deletions docs/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -472,34 +472,28 @@ Rerank documents based on a query.

## Tool Calling Support

OpenArc supports OpenAI-compatible tool calling. Tools are parsed from model output using regex pattern matching for JSON objects containing `name` and `arguments` fields.
OpenArc supports OpenAI-compatible tool calling. Tools are parsed from model output using Hermes-style `<tool_call>...</tool_call>` tags containing JSON with `name` and `arguments` fields.

Tool calls are detected in streaming and non-streaming modes:
- **Streaming**: Tool calls are detected incrementally and streamed as structured chunks
- **Non-streaming**: Tool calls are parsed from the final output

### Parser Implementation

The `parse_tool_calls()` function searches for JSON objects in the model's text output and converts them to OpenAI-compatible tool call format.
The `parse_tool_calls()` function extracts payloads from `<tool_call>...</tool_call>` tags in the model's text output and converts them to OpenAI-compatible tool call format.

**Input Format (Model Output):**

The parser expects JSON objects embedded in the text with the following structure:
The parser expects Hermes-style tagged payloads with the following structure:

```json
{
"name": "function_name",
"arguments": {
"arg1": "value1",
"arg2": "value2"
}
}
<tool_call>{"name":"function_name","arguments":{"arg1":"value1","arg2":"value2"}}</tool_call>
```

**Input to the parser from a model:**

```
The user wants to know the weather. {"name": "get_weather", "arguments": {"location": "San Francisco", "units": "celsius"}} I'll check that for you.
The user wants to know the weather. <tool_call>{"name":"get_weather","arguments":{"location":"San Francisco","units":"celsius"}}</tool_call> I'll check that for you.
```

**Output Format (OpenAI-Compatible):**
Expand All @@ -521,8 +515,9 @@ Parser returns a list of tool call objects in OpenAI format:

**Parser Behavior:**

- Searches for JSON objects using regex pattern: `\{(?:[^{}]|(?:\{[^{}]*\}))*\}`
- Validates that each JSON object contains both `name` and `arguments` fields
- Extracts JSON payloads from `<tool_call>...</tool_call>` tags
- Supports an EOS fallback when a `<tool_call>` start tag appears without a closing `</tool_call>`
- Validates that each payload contains both `name` and `arguments` fields
- Generates unique IDs in format `call_{24-char-hex}`
- Converts `arguments` to JSON string (required by OpenAI format)
- Returns `None` if no valid tool calls are found
Expand Down
20 changes: 1 addition & 19 deletions src/server/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
import json
import logging
import os
import re
import time
import traceback
import uuid
Expand Down Expand Up @@ -214,24 +213,7 @@ def parse_tool_calls(text: str) -> Optional[List[Dict[str, Any]]]:
if tool_calls:
return tool_calls

# Backward compatibility for plain JSON tool call outputs without tags.
pattern = r"\{(?:[^{}]|(?:\{[^{}]*\}))*\}"
for match in re.findall(pattern, text, re.DOTALL):
try:
data = json.loads(match)
if isinstance(data, dict) and "name" in data and "arguments" in data:
tool_calls.append({
"id": f"call_{uuid.uuid4().hex[:24]}",
"type": "function",
"function": {
"name": str(data.get("name", "")),
"arguments": _format_tool_call_arguments(data.get("arguments", {})),
},
})
except json.JSONDecodeError:
continue

return tool_calls if tool_calls else None
return None

#===============================================================#
# OpenArc internal
Expand Down
8 changes: 8 additions & 0 deletions src/tests/test_tool_call_parser_unit.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,14 @@ def test_parse_tool_calls_supports_missing_closing_tag_until_eos() -> None:
assert json.loads(tool_calls[0]["function"]["arguments"]) == {"query": "vLLM"}


def test_parse_tool_calls_rejects_plain_json_without_tool_call_tags() -> None:
text = '{"name":"search","arguments":{"query":"legacy"}}'

tool_calls = server_main.parse_tool_calls(text)

assert tool_calls is None


@pytest.mark.asyncio
async def test_openai_chat_completions_non_streaming_tool_calls(monkeypatch: pytest.MonkeyPatch) -> None:
class _Workers:
Expand Down