[Feature]: Support structured output and tool call together

[Wendong-Fan]

🚀 The feature, motivation and pitch

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

as discussed in: https://www.reddit.com/r/LocalLLaMA/comments/1h2y7ys/why_can_you_not_use_structured_output_and_tool/

request support for tool execution followed by structured response output, similar to how OpenAI handles function calls (Tool Calls) and Schema outputs in the gpt-4o-mini API.

OpenAI supports this capability through a workflow where:

The model can first execute one or more tool calls to perform calculations or retrieve information
After receiving the results of these tool calls, the model can then produce a final structured response conforming to a predefined schema

Example Implementation from OpenAI

Input to OpenAI
User query: "current age of University of Oxford is 928"
Available tools: add, subtract, multiply, divide, round
Response format: Schema with current_age and calculated_age fields

Execution flow:

1. Model analyzes query and determines it needs to call the add function
2. Model executes: add(928, 1) to calculate the next year's age
3. Function returns: 929
4. Model produces final structured response:
   {"current_age":"928","calculated_age":"929"}

=================Input messages to OpenAI=================
[{'role': 'system', 'content': 'You are a helpful assistant. You must call tools'}, {'role': 'user', 'content': 'current age of University of Oxford is 928, '}]
=================Input config to OpenAI=================
{'temperature': 0.0, 'response_format': <class '__main__.Schema'>, 'tools': [{'type': 'function', 'function': {'name': 'add', 'description': 'Adds two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The first number to be added.'}, 'b': {'type': 'number', 'description': 'The second number to be added.'}}, 'required': ['a', 'b'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'sub', 'description': 'Do subtraction between two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The minuend in subtraction.'}, 'b': {'type': 'number', 'description': 'The subtrahend in subtraction.'}}, 'required': ['a', 'b'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'multiply', 'description': 'Multiplies two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The multiplier in the multiplication.'}, 'b': {'type': 'number', 'description': 'The multiplicand in the multiplication.'}, 'decimal_places': {'type': 'integer', 'description': 'The number of decimal\nplaces to round to. Defaults to 2.'}}, 'required': ['a', 'b', 'decimal_places'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'divide', 'description': 'Divides two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The dividend in the division.'}, 'b': {'type': 'number', 'description': 'The divisor in the division.'}, 'decimal_places': {'type': 'integer', 'description': 'The number of\ndecimal places to round to. Defaults to 2.'}}, 'required': ['a', 'b', 'decimal_places'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'round', 'description': 'Rounds a number to a specified number of decimal places.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The number to be rounded.'}, 'decimal_places': {'type': 'integer', 'description': 'The number of decimal places\nto round to. Defaults to 0.'}}, 'required': ['a', 'decimal_places'], 'type': 'object', 'additionalProperties': False}}}]}
=================Output from OpenAI=================
ParsedChatCompletion[Schema](id='chatcmpl-BJpADooa7xu5xJHJNjCJ1Je4R6Oa6', choices=[ParsedChoice[Schema](finish_reason='tool_calls', index=0, logprobs=None, message=ParsedChatCompletionMessage[Schema](content=None, refusal=None, role='assistant', annotations=[], audio=None, function_call=None, tool_calls=[ParsedFunctionToolCall(id='call_WmGiI0q8YRNojvxAQIbCg1EH', function=ParsedFunction(arguments='{"a":928,"b":1}', name='add', parsed_arguments={'a': 928, 'b': 1}), type='function')], parsed=None))], created=1744064605, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_b376dfbbd5', usage=CompletionUsage(completion_tokens=18, prompt_tokens=351, total_tokens=369, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))
***************add function called*********************
=================Input messages to OpenAI=================
[{'role': 'system', 'content': 'You are a helpful assistant. You must call tools'}, {'role': 'user', 'content': 'current age of University of Oxford is 928, '}, {'role': 'assistant', 'content': '', 'tool_calls': [{'id': 'call_PaitDpihu1REKPNJnx1qtSSe', 'type': 'function', 'function': {'name': 'add', 'arguments': '{"a": 928, "b": 1}'}}]}, {'role': 'tool', 'content': '929', 'tool_call_id': 'call_PaitDpihu1REKPNJnx1qtSSe'}]
=================Input config to OpenAI=================
{'temperature': 0.0, 'response_format': <class '__main__.Schema'>, 'tools': [{'type': 'function', 'function': {'name': 'add', 'description': 'Adds two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The first number to be added.'}, 'b': {'type': 'number', 'description': 'The second number to be added.'}}, 'required': ['a', 'b'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'sub', 'description': 'Do subtraction between two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The minuend in subtraction.'}, 'b': {'type': 'number', 'description': 'The subtrahend in subtraction.'}}, 'required': ['a', 'b'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'multiply', 'description': 'Multiplies two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The multiplier in the multiplication.'}, 'b': {'type': 'number', 'description': 'The multiplicand in the multiplication.'}, 'decimal_places': {'type': 'integer', 'description': 'The number of decimal\nplaces to round to. Defaults to 2.'}}, 'required': ['a', 'b', 'decimal_places'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'divide', 'description': 'Divides two numbers.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The dividend in the division.'}, 'b': {'type': 'number', 'description': 'The divisor in the division.'}, 'decimal_places': {'type': 'integer', 'description': 'The number of\ndecimal places to round to. Defaults to 2.'}}, 'required': ['a', 'b', 'decimal_places'], 'type': 'object', 'additionalProperties': False}}}, {'type': 'function', 'function': {'name': 'round', 'description': 'Rounds a number to a specified number of decimal places.', 'strict': True, 'parameters': {'properties': {'a': {'type': 'number', 'description': 'The number to be rounded.'}, 'decimal_places': {'type': 'integer', 'description': 'The number of decimal places\nto round to. Defaults to 0.'}}, 'required': ['a', 'decimal_places'], 'type': 'object', 'additionalProperties': False}}}]}
=================Output from OpenAI=================
ParsedChatCompletion[Schema](id='chatcmpl-BJpAEN4GchSUheJzl3bKeVKXTuD11', choices=[ParsedChoice[Schema](finish_reason='stop', index=0, logprobs=None, message=ParsedChatCompletionMessage[Schema](content='{"current_age":"928","calculated_age":"929"}', refusal=None, role='assistant', annotations=[], audio=None, function_call=None, tool_calls=None, parsed=Schema(current_age='928', calculated_age='929')))], created=1744064606, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_b376dfbbd5', usage=CompletionUsage(completion_tokens=17, prompt_tokens=376, total_tokens=393, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))
{"current_age":"928","calculated_age":"929"}

Related resources

No response

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[gothaleshubham]

I am also facing same issue that tool_calling and structure output not working together.

[andysalerno]

This is a pretty big issue because the OpenAI Agents SDK makes heavy use of this.

If you define an agent like this:

class ResearchComplete(BaseModel):
    handoff_message: str

....

def create_agent():
    return Agent(
        name="ResearchAgent",
        model=get_model(),
        tools=[],
        output_type=ResearchComplete,
        mcp_servers=[mcp_server],
        model_settings=ModelSettings(
            temperature=temp,
        ),
        instructions=_create_prompt(
            cur_date,
        ),
    )

Then it ends up making a request like this (messages omitted):

{
    "model": "hosted_vllm/model",
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "search_web",
                "description": "\nSearch Google for the given query. Returns the search results (including snippet and url) in JSON format.\nUseful for looking up: current events, up-to-date information, and general knowledge across the web.\n\nArgs:\n    query: The query to search for. Remember, this is a google query, so write your query as you would in Google.\n",
                "parameters": {
                    "properties": {
                        "query": {
                            "title": "Query",
                            "type": "string"
                        }
                    },
                    "required": [
                        "query"
                    ],
                    "title": "search_webArguments",
                    "type": "object"
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "visit_url",
                "description": "\nVisits (scrapes) the given URL and returns the text content of the page.\nUseful for looking up current events, or visiting urls you found in your research.\n\nArgs:\n    url: The url to visit (scrape).\n",
                "parameters": {
                    "properties": {
                        "url": {
                            "title": "Url",
                            "type": "string"
                        }
                    },
                    "required": [
                        "url"
                    ],
                    "title": "scrapeArguments",
                    "type": "object"
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "execute_python_code",
                "description": "\nExecutes Python code and returns the output from stdout.\nMultiple lines are allowed, including complex scripting.\nIMPORTANT: you ONLY see the results from stdout, so you MUST print() the result you want to see.\nSimply placing the result on the last line will not work.\n\nArgs:\n    code: The Python code to execute, which must include a print statement to output the result.\n",
                "parameters": {
                    "properties": {
                        "code": {
                            "title": "Code",
                            "type": "string"
                        }
                    },
                    "required": [
                        "code"
                    ],
                    "title": "execute_python_codeArguments",
                    "type": "object"
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "search_wikipedia",
                "description": "\n\"Search Wikipedia for the given query and return the results. Includes multiple document titles and body content. \"\n\"Great for looking up people, places, and things. Less great (but still good) for current events.\"\n",
                "parameters": {
                    "properties": {
                        "query": {
                            "title": "Query",
                            "type": "string"
                        }
                    },
                    "required": [
                        "query"
                    ],
                    "title": "search_wikipediaArguments",
                    "type": "object"
                }
            }
        }
    ],
    "temperature": 0.0,
    "top_p": null,
    "frequency_penalty": null,
    "presence_penalty": null,
    "max_tokens": null,
    "tool_choice": null,
    "response_format": {
        "type": "json_schema",
        "json_schema": {
            "name": "final_output",
            "strict": true,
            "schema": {
                "properties": {
                    "handoff_message": {
                        "title": "Handoff Message",
                        "type": "string"
                    }
                },
                "required": [
                    "handoff_message"
                ],
                "title": "ResearchComplete",
                "type": "object",
                "additionalProperties": false
            }
        }
    },
    "parallel_tool_calls": null,
    "stream": false,
    "stream_options": null,
    "reasoning_effort": null,
    "extra_headers": {
        "User-Agent": "Agents/Python 0.2.5"
    },
    "base_url": "http://localhost:8000/v1"
}

Since VLLM doesn't understand this, it will always follow the json_schema provided and never call any tools.

[bbartels]

We are running into this issue as well, would be useful if this was supported!

[mladjan-gadzic]

i've created draft pr #29612 for this and i'm looking for early feedback. thanks!