Skip to content

Commit 90ebe5e

Browse files
EricLBuehlerEricLBuehler
committed
Update docs
1 parent 1e1f8c5 commit 90ebe5e

File tree

1 file changed

+116
-110
lines changed

1 file changed

+116
-110
lines changed

‎docs/MCP.md‎

Lines changed: 116 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -1,93 +1,60 @@
11
# MCP protocol support
22

3-
`mistralrs-server` can serve **MCP (Model Control Protocol)** traffic next to the regular OpenAI-compatible HTTP interface.
3+
`mistralrs-server` can serve **MCP (Model Control Protocol)** traffic next to the regular OpenAI-compatible HTTP interface!
44

55
MCP is an open, tool-based protocol that lets clients interact with models through structured *tool calls* instead of free-form HTTP routes.
66

7-
Under the hood the server uses [`rust-mcp-sdk`](https://crates.io/crates/rust-mcp-sdk) and exposes a single tool called **`chat`** that mirrors the behaviour of the `/v1/chat/completions` endpoint.
7+
Under the hood the server uses [`rust-mcp-sdk`](https://crates.io/crates/rust-mcp-sdk) and exposes tools based on the supported modalities of the loaded model.
88

9-
---
9+
Exposed tools:
1010

11-
## 1. Building
11+
| Tool | Minimum `input` -> `output` modalities | Description |
12+
| -- | -- | -- |
13+
| `chat` | | `Text` -> `Text` | Wraps the OpenAI `/v1/chat/completions` endpoint. |
1214

13-
Support for MCP is compiled in by default because the workspace enables the `server` and `hyper-server` features of `rust-mcp-sdk`.
14-
When you only compile the `mistralrs-server` crate outside the workspace enable the `mcp-server` Cargo feature manually:
1515

16-
```bash
17-
cargo build -p mistralrs-server --release --features "mcp-server"
18-
```
16+
---
17+
18+
## ToC
19+
- [MCP protocol support](#mcp-protocol-support)
20+
- [ToC](#toc)
21+
- [Running](#running)
22+
- [Check if it's working](#check-if-its-working)
23+
- [Example clients](#example-clients)
24+
- [Rust](#rust)
25+
- [Python](#python)
26+
- [HTTP](#http)
27+
- [Limitations](#limitations)
1928

20-
## 2. Running
29+
---
30+
31+
## Running
2132

2233
Start the normal HTTP server and add the `--mcp-port` flag to spin up an MCP server on a separate port:
2334

2435
```bash
2536
./target/release/mistralrs-server \
2637
--port 1234 # OpenAI compatible HTTP API
27-
--mcp-port 4321 # MCP protocol endpoint (SSE over HTTP)
38+
--mcp-port 4321 # MCP protocol endpoint (Streamable HTTP)
2839
plain -m mistralai/Mistral-7B-Instruct-v0.3
2940
```
3041

31-
* `--mcp-port` takes precedence over `--port` – you can run the HTTP and MCP servers on totally independent ports or omit `--port` when you only need MCP.*
32-
33-
## 3. Capabilities announced to clients
34-
35-
At start-up the MCP handler advertises the following `InitializeResult` (abridged):
36-
37-
```jsonc
38-
{
39-
"server_info": { "name": "mistralrs", "version": "<crate-version>" },
40-
"protocol_version": "2025-03-26", // latest spec version from rust-mcp-sdk
41-
"instructions": "use tool 'chat'",
42-
"capabilities": {
43-
"tools": {}
44-
}
45-
}
46-
```
47-
48-
Only one tool is currently exposed:
49-
50-
| tool | description |
51-
|------|------------------------------------------------------|
52-
| `chat` | Wraps the OpenAI `/v1/chat/completions` endpoint. |
42+
## Check if it's working
5343

54-
## 4. Calling the `chat` tool
44+
Run this `curl` command to check the available tools:
5545

56-
Clients send a [`CallToolRequest`](https://docs.rs/rust-mcp-schema/latest/rust_mcp_schema/struct.CallToolRequest.html) event where `params.name` is `"chat"` and `params.arguments` contains a standard MCP [`CreateMessageRequest`](https://docs.rs/rust-mcp-schema/latest/rust_mcp_schema/struct.CreateMessageRequest.html).
57-
58-
Example request (sent as SSE `POST /mcp/stream` or via the convenience helpers in `rust-mcp-sdk`):
59-
60-
```jsonc
61-
{
62-
"kind": "callToolRequest",
63-
"id": "123",
64-
"params": {
65-
"name": "chat",
66-
"arguments": {
67-
"model": "mistralai/Mistral-7B-Instruct-v0.3",
68-
"messages": [
69-
{ "role": "user", "content": "Explain Rust ownership." }
70-
]
71-
}
72-
}
73-
}
7446
```
75-
76-
The response is a `CallToolResult` event whose `content` array contains a single `TextContent` item with the assistant response.
77-
78-
```jsonc
79-
{
80-
"kind": "callToolResult",
81-
"id": "123",
82-
"content": [
83-
{ "type": "text", "text": "Rust’s ownership system ..." }
84-
]
85-
}
47+
curl -X POST http://localhost:4321/mcp \
48+
-H "Content-Type: application/json" \
49+
-d '{
50+
"jsonrpc": "2.0",
51+
"id": 2,
52+
"method": "tools/list",
53+
"params": {}
54+
}'
8655
```
8756

88-
Error cases are mapped to `CallToolError` with `is_error = true`.
89-
90-
## 5. Example clients
57+
## Example clients
9158

9259
### Rust
9360

@@ -109,7 +76,7 @@ impl rust_mcp_sdk::mcp_client::ClientHandler for Handler {}
10976
#[tokio::main]
11077
async fn main() -> Result<()> {
11178
let transport = ClientSseTransport::new(
112-
"http://localhost:4321/mcp/stream",
79+
"http://localhost:4321/mcp",
11380
ClientSseTransportOptions::default(),
11481
)?;
11582

@@ -141,56 +108,95 @@ async fn main() -> Result<()> {
141108
### Python
142109

143110
```py
144-
import json
145-
import requests
146-
from sseclient import SSEClient
147-
148-
payload = {
149-
"kind": "callToolRequest",
150-
"id": "123",
151-
"params": {
152-
"name": "chat",
153-
"arguments": {
154-
"model": "mistralai/Mistral-7B-Instruct-v0.3",
155-
"messages": [
156-
{"role": "user", "content": "Explain Rust ownership."}
157-
],
158-
},
159-
},
160-
}
161-
162-
resp = requests.post(
163-
"http://localhost:4321/mcp/stream",
164-
headers={"Content-Type": "application/json"},
165-
data=json.dumps(payload),
166-
stream=True,
167-
)
168-
for event in SSEClient(resp):
169-
print(event.data)
111+
import asyncio
112+
from mcp import ClientSession
113+
from mcp.client.streamable_http import streamablehttp_client
114+
115+
SERVER_URL = "http://localhost:4321/mcp"
116+
117+
async def main() -> None:
118+
async with streamablehttp_client(SERVER_URL) as (read, write, _):
119+
async with ClientSession(read, write) as session:
120+
121+
# --- INITIALIZE ---
122+
init_result = await session.initialize()
123+
print("Server info:", init_result.serverInfo)
124+
125+
# --- LIST TOOLS ---
126+
tools = await session.list_tools()
127+
print("Available tools:", [t.name for t in tools.tools])
128+
129+
# --- CALL TOOL ---
130+
resp = await session.call_tool(
131+
"chat",
132+
arguments={
133+
"messages": [
134+
{"role": "user", "content": "Hello MCP 👋"},
135+
{"role": "assistant", "content": "Hi there!"}
136+
],
137+
"maxTokens": 50,
138+
"temperature": 0.7,
139+
},
140+
)
141+
# resp.content is a list[CallToolResultContentItem]; extract text parts
142+
text = "\n".join(c.text for c in resp.content if c.type == "text")
143+
print("Model replied:", text)
144+
145+
if __name__ == "__main__":
146+
asyncio.run(main())
170147
```
171148

172149
### HTTP
173150

151+
**Call a tool:**
152+
```bash
153+
curl -X POST http://localhost:4321/mcp \
154+
-H "Content-Type: application/json" \
155+
-d '{
156+
"jsonrpc": "2.0",
157+
"id": 3,
158+
"method": "tools/call",
159+
"params": {
160+
"name": "chat",
161+
"arguments": {
162+
"messages": [
163+
{ "role": "system", "content": "You are a helpful assistant." },
164+
{ "role": "user", "content": "Hello, what’s the time?" }
165+
],
166+
"maxTokens": 50,
167+
"temperature": 0.7
168+
}
169+
}
170+
}'
171+
```
172+
173+
**Initialize:**
174+
```bash
175+
curl -X POST http://localhost:4321/mcp \
176+
-H "Content-Type: application/json" \
177+
-d '{
178+
"jsonrpc": "2.0",
179+
"id": 1,
180+
"method": "initialize",
181+
"params": {}
182+
}'
183+
```
184+
185+
**List tools:**
174186
```bash
175-
curl -N -X POST http://localhost:4321/mcp/stream \
176-
-H 'Content-Type: application/json' \
177-
-d '{
178-
"kind": "callToolRequest",
179-
"id": "123",
180-
"params": {
181-
"name": "chat",
182-
"arguments": {
183-
"model": "mistralai/Mistral-7B-Instruct-v0.3",
184-
"messages": [{"role": "user", "content": "Explain Rust ownership."}]
185-
}
186-
}
187-
}'
187+
curl -X POST http://localhost:4321/mcp \
188+
-H "Content-Type: application/json" \
189+
-d '{
190+
"jsonrpc": "2.0",
191+
"id": 2,
192+
"method": "tools/list",
193+
"params": {}
194+
}'
188195
```
189196

190-
## 6. Limitations & future work
197+
## Limitations
191198

192-
• Only synchronous, single-shot requests are supported right now.
193-
• Streaming responses (`partialCallToolResult`) are not yet implemented.
194-
• No authentication layer is provided – run the MCP port behind a reverse proxy if you need auth.
199+
- Streaming requests are not implemented.
200+
- No authentication layer is provided – run the MCP port behind a reverse proxy if you need auth.
195201

196202
Contributions to extend MCP coverage (streaming, more tools, auth hooks) are welcome!

0 commit comments

Comments
 (0)