Testing using OpenCode

[TomLucidor]

Mentioned here #33 (comment)

The idea is that for integration to work, assuming that OpenCode is functional with OpenAI-compatible API and OpenRouter, we can use it to test and see if vLLM-MLX is functional or not by injecting some sort of "Hello World" message. If </think> manages to leak out (failing to turn off thinking, parser error), we can assume something is broken.
Similar idea can be said for tool calling e.g. list test directory, write new file X, modify a few lines of file Y. If the tool parsing is broken it can be tested through observing the text files that were modified, and the changes in the test directory.

[TomLucidor]

@waybarrios some background info for testing
For ~/.config/opencode/oh-my-opencode.json

{
  "$schema": "https://raw.githubusercontent.com/code-yeongyu/oh-my-opencode/master/assets/oh-my-opencode.schema.json",
  "agents": {
    "sisyphus": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "librarian": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "explore": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "oracle": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "multimodal-looker": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "prometheus": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "metis": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "momus": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "atlas": {
      "model": "MLX/Ring-Mini-Linear"
    }
  },
  "categories": {
    "visual-engineering": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "ultrabrain": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "artistry": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "quick": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "unspecified-low": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "unspecified-high": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "writing": {
      "model": "MLX/Ring-Mini-Linear"
    },
    "deep": {
      "model": "MLX/Ring-Mini-Linear"
    }
  }
}

For opencode.json within the project folder

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "MLX": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "vLLM-MLX",
      "options": {
        "baseURL": "http://127.0.0.1:8000/v1"
      },
      "models": {
        "Ring-Mini-Linear": {
          "id": "mlx-community/Ring-mini-linear-2.0-6bit",
          "name": "Ring-Mini-Linear"
        }
      }
    }
  }
}

[janhilgard]

Thanks @TomLucidor for the detailed testing proposal and OpenCode config! This is a great approach for integration testing.

Current status

Both areas you identified are now implemented and working:

1. Reasoning parser (</think> tag handling) ✅

PR #33 (merged Feb 8) added:

• --reasoning-parser qwen3|deepseek_r1 CLI flag for proper <think> content extraction
• Think tag stripping in tool parsers — prevents </think> leaking into output even when models mix reasoning with tool calls (the Ring-Mini-Linear issue you flagged)

2. Tool calling ✅

PR #46 (merged Feb 8) added streaming tool call parsing with hermes and qwen parsers, plus fixes for multi-turn tool calling. Tested working with Qwen3-Next-80B and Qwen3-Coder-Next via OpenCode and Claude Code.

Recommended OpenCode config for current version

For models with reasoning (Qwen3, DeepSeek-R1, Ring-Mini-Linear):

Server launch:

vllm-mlx serve mlx-community/Qwen3-Next-80B-A3B-Instruct-6bit \
    --port 1239 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "MLX": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "vLLM-MLX",
      "options": {
        "baseURL": "http://127.0.0.1:1239/v1"
      },
      "models": {
        "Qwen3-Next-80B": {
          "id": "mlx-community/Qwen3-Next-80B-A3B-Instruct-6bit",
          "name": "Qwen3-Next-80B"
        }
      }
    }
  }
}

Next steps

Keeping this open as a tracking issue for integration test development. The manual testing approach you described (hello world for reasoning, file ops for tool calling) could be formalized into a test suite. Would welcome contributions on that front!

[enryold]

Performance observation: vllm-mlx vs LM Studio with OpenCode

I've been testing vllm-mlx with OpenCode and noticed that LM Studio feels noticeably faster for this use case.

After investigating, the root cause seems to be OpenCode's request pattern: it sends 2 requests at startup, then 1 request at a time sequentially. Since vllm-mlx is essentially single-request most of the time, the Python overhead (all the processing layers before the actual MLX engine call) becomes a bottleneck compared to LM Studio, which has a more lightweight native path to the inference engine.

Question: Is this 1-request-at-a-time behavior the default/standard OpenCode configuration, or can it be tweaked to send more parallel requests? If parallelism could be increased, vllm-mlx's batching and serving infrastructure would have a chance to actually pay off.

On the positive side: Tool calling worked like a charm with vllm-mlx — no issues there at all.

Side note on LM Studio: While LM Studio was faster for this workflow, I tested with Qwen3-Coder-30B-A3B and LM Studio's KV cache quantization didn't work as expected with that model — so it's not without its own issues.

[janhilgard]

@enryold Great feedback — the single-request Python overhead observation is fair for a cold first request. But in real agentic workflows (OpenCode, Claude Code, OpenClaw), vllm-mlx's optimizations stack multiplicatively and more than compensate:

1. MTP (Multi-Token Prediction) — 1.4–1.8x throughput

PR #82 adds MTP speculative decoding for models with built-in MTP heads (Qwen3-Next, Nemotron-Nano):

Mode	tok/s	Speedup
Baseline	55	1.0x
MTP verified	79	1.43x
MTP optimistic	97	1.76x

(Qwen3-Next-80B-A3B-6bit, M3 Ultra 256GB)

2. Prefix caching — 26–40x TTFT reduction

This is the killer feature for agentic workflows. Tool-calling clients send the entire conversation history with every request — vllm-mlx caches the KV state and skips re-processing:

Prompt size	Without cache	With cache	Improvement
50K tokens	30–50s	0.3–1.7s	~30x
100K tokens	60–90s	1.6–2.1s	~40x

In a real Claude Code session (17 tool-call requests): 93.7% tokens saved, TTFT 1–3s consistently. A 108K-token prompt served from cache in 2.1s instead of ~90s.

3. KV cache quantization — 3.5x memory savings

PR #62 adds 8-bit KV cache quantization, reducing cache memory ~3.5x. This means more concurrent requests fit in memory, and the prefix cache can hold more entries before eviction.

4. Continuous batching + MTP at scale

With all optimizations enabled and multiple concurrent requests:

Parallel requests	Aggregate tok/s
1	70
4	109
8	145
16	203

The real comparison

LM Studio has lower cold-start overhead for a single isolated request — that's true. But agentic workflows are not isolated requests. They're long multi-turn conversations where:

1. Prefix caching saves 90%+ of prefill computation (the actual bottleneck)
2. MTP speeds up generation 1.4–1.8x
3. Continuous batching handles the 2+ startup requests OpenCode sends concurrently

For the Qwen3-Coder-30B-A3B model you tested — try launching with --enable-mtp and after the first request, every subsequent turn in the same conversation will hit prefix cache. The difference vs LM Studio should be very noticeable on turn 3+.

vllm-mlx serve mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit \
    --port 8000 \
    --continuous-batching \
    --cache-memory-mb 16384 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --enable-mtp

[waybarrios]

I've also noticed some slowness when using open source lately. I'm doing some research and hope to have some feedback in the next few days.

…

On Fri, Feb 13, 2026 at 4:17 PM Jan Hilgard ***@***.***> wrote: *janhilgard* left a comment (waybarrios/vllm-mlx#34) <#34 (comment)> @enryold <https://github.com/enryold> Great feedback — the single-request Python overhead observation is fair for a cold first request. But in real agentic workflows (OpenCode, Claude Code, OpenClaw), vllm-mlx's optimizations stack multiplicatively and more than compensate: 1. MTP (Multi-Token Prediction) — 1.4–1.8x throughput PR #82 <#82> adds MTP speculative decoding for models with built-in MTP heads (Qwen3-Next, Nemotron-Nano): Mode tok/s Speedup Baseline 55 1.0x MTP verified 79 *1.43x* MTP optimistic 97 *1.76x* *(Qwen3-Next-80B-A3B-6bit, M3 Ultra 256GB)* 2. Prefix caching — 26–40x TTFT reduction This is the killer feature for agentic workflows. Tool-calling clients send the entire conversation history with every request — vllm-mlx caches the KV state and skips re-processing: Prompt size Without cache With cache Improvement 50K tokens 30–50s 0.3–1.7s *~30x* 100K tokens 60–90s 1.6–2.1s *~40x* In a real Claude Code session (17 tool-call requests): *93.7% tokens saved*, TTFT 1–3s consistently. A 108K-token prompt served from cache in *2.1s* instead of ~90s. 3. KV cache quantization — 3.5x memory savings PR #62 <#62> adds 8-bit KV cache quantization, reducing cache memory ~3.5x. This means more concurrent requests fit in memory, and the prefix cache can hold more entries before eviction. 4. Continuous batching + MTP at scale With all optimizations enabled and multiple concurrent requests: Parallel requests Aggregate tok/s 1 70 4 109 8 145 16 *203* The real comparison LM Studio has lower cold-start overhead for a single isolated request — that's true. But agentic workflows are *not* isolated requests. They're long multi-turn conversations where: 1. *Prefix caching* saves 90%+ of prefill computation (the actual bottleneck) 2. *MTP* speeds up generation 1.4–1.8x 3. *Continuous batching* handles the 2+ startup requests OpenCode sends concurrently For the Qwen3-Coder-30B-A3B model you tested — try launching with --enable-mtp and after the first request, every subsequent turn in the same conversation will hit prefix cache. The difference vs LM Studio should be very noticeable on turn 3+. vllm-mlx serve mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit \ --port 8000 \ --continuous-batching \ --cache-memory-mb 16384 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --enable-mtp — Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABT24TEYLZJIJCQEPTXZCMT4LY5NLAVCNFSM6AAAAACTYULHD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQOJZGUZDGNRXG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[waybarrios]

I think it would be interesting to do some profiling here and see what causes the delay. And then based on that, we could purpose an elegant solution to patch this thing.

…

On Fri, Feb 13, 2026 at 4:20 PM Way Barrios ***@***.***> wrote: I've also noticed some slowness when using open source lately. I'm doing some research and hope to have some feedback in the next few days. On Fri, Feb 13, 2026 at 4:17 PM Jan Hilgard ***@***.***> wrote: > *janhilgard* left a comment (waybarrios/vllm-mlx#34) > <#34 (comment)> > > @enryold <https://github.com/enryold> Great feedback — the > single-request Python overhead observation is fair for a cold first > request. But in real agentic workflows (OpenCode, Claude Code, OpenClaw), > vllm-mlx's optimizations stack multiplicatively and more than compensate: > 1. MTP (Multi-Token Prediction) — 1.4–1.8x throughput > > PR #82 <#82> adds MTP > speculative decoding for models with built-in MTP heads (Qwen3-Next, > Nemotron-Nano): > Mode tok/s Speedup > Baseline 55 1.0x > MTP verified 79 *1.43x* > MTP optimistic 97 *1.76x* > > *(Qwen3-Next-80B-A3B-6bit, M3 Ultra 256GB)* > 2. Prefix caching — 26–40x TTFT reduction > > This is the killer feature for agentic workflows. Tool-calling clients > send the entire conversation history with every request — vllm-mlx caches > the KV state and skips re-processing: > Prompt size Without cache With cache Improvement > 50K tokens 30–50s 0.3–1.7s *~30x* > 100K tokens 60–90s 1.6–2.1s *~40x* > > In a real Claude Code session (17 tool-call requests): *93.7% tokens > saved*, TTFT 1–3s consistently. A 108K-token prompt served from cache in > *2.1s* instead of ~90s. > 3. KV cache quantization — 3.5x memory savings > > PR #62 <#62> adds 8-bit KV > cache quantization, reducing cache memory ~3.5x. This means more concurrent > requests fit in memory, and the prefix cache can hold more entries before > eviction. > 4. Continuous batching + MTP at scale > > With all optimizations enabled and multiple concurrent requests: > Parallel requests Aggregate tok/s > 1 70 > 4 109 > 8 145 > 16 *203* The real comparison > > LM Studio has lower cold-start overhead for a single isolated request — > that's true. But agentic workflows are *not* isolated requests. They're > long multi-turn conversations where: > > 1. *Prefix caching* saves 90%+ of prefill computation (the actual > bottleneck) > 2. *MTP* speeds up generation 1.4–1.8x > 3. *Continuous batching* handles the 2+ startup requests OpenCode > sends concurrently > > For the Qwen3-Coder-30B-A3B model you tested — try launching with > --enable-mtp and after the first request, every subsequent turn in the > same conversation will hit prefix cache. The difference vs LM Studio should > be very noticeable on turn 3+. > > vllm-mlx serve mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit \ > --port 8000 \ > --continuous-batching \ > --cache-memory-mb 16384 \ > --enable-auto-tool-choice \ > --tool-call-parser hermes \ > --enable-mtp > > — > Reply to this email directly, view it on GitHub > <#34 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABT24TEYLZJIJCQEPTXZCMT4LY5NLAVCNFSM6AAAAACTYULHD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQOJZGUZDGNRXG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[Lerg]

I'm testing LM Studio with MLX backend. Qwen3-coder-next 4bit, max context, M4 Max 64GB.
Tried codex, claude, opencode and qwen. qwen seems to be performing the best overall so far (tried anthropic api).
The main bottleneck is the cache trimming. LM Studio reports that the cache is not trimmable when the coding agents wants to clear a few thousand tokens, LM Studio clears the entire cache instead which results in frequent and long cache rebuilds.

Are you guys also seeing this behavior with vLLM-MLX or is this LM Studio only problem?