Speculative Decoding (MTP) Doesn't Help Japanese Small Talk — Acceptance 36% vs 71%, Measured

TL;DR

• We enabled the native MTP (Multi-Token Prediction) head on ThinkingCap-Qwen3.6-27B (NVFP4) in vLLM and measured single-stream decode speed
• MTP draft acceptance was 71% for English code generation, 36% for Japanese casual chat. Speedup was 1.85x versus 1.12x — a night-and-day difference depending on workload
• "A 9x active-parameter gap between a MoE and a dense model should mean 9x slower" turns out to be wrong. The measured gap was 2.5x, and the gap between 9x and 2.5x is fully explained by bandwidth math (quantization narrows 9x to 5x, and fixed overhead further narrows 5x to 2.5x)
• For real-time Japanese conversation like voice chat, don't design around speculative decoding paying off. For agentic/coding workloads, it's nearly free and gets you close to 2x

Background: switching from MoE to dense made speed a problem

I was evaluating replacing a running local LLM — a 35B-A3B MoE with 3B active parameters — with a dense 27B model. Instruction-following and agentic performance were better on the dense side. The problem was decode speed: real-time voice chat runs on the same engine, so the constraint was "if conversation gets slower, this is a non-starter."

That's when the candidate model's bundled MTP (Multi-Token Prediction) head for speculative decoding became interesting. The question this post answers: can speculation make up for dense's speed handicap?

Environment and methodology

• GPU: RTX PRO 6000 Blackwell (96GB)
• vLLM 0.24.0 / Model: ThinkingCap-Qwen3.6-27B-NVFP4 (weights ~19.7GB + MTP head bf16 0.85GB)
• --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' / --kv-cache-dtype fp8
• Comparison target: Agents-A1 (35B-A3B MoE, fp8, no speculation)
• Single stream, streaming mode, measuring TTFT and decode tok/s (completion_tokens / elapsed time after the first token)
• Acceptance computed from the delta in vLLM's /metrics:

acceptance = Δ vllm:spec_decode_num_accepted_tokens_total
           / Δ vllm:spec_decode_num_draft_tokens_total

Two scenarios: ja_chat (persona-carrying Japanese casual conversation, 512-token cap) and code_agent (English coding task, 2048-token cap).

Results

Decode tok/s (median)	MoE 35B-A3B fp8	dense 27B plain	dense 27B + MTP	MTP acceptance
Japanese casual chat	177	72.0	80.5 (x1.12)	36.1% (92/255)
English code generation	177	70.7	131.0 (x1.85)	71.4% (2795/3912)

TTFT was 50-100ms across every configuration, with no meaningful difference.

Plain decode is flat regardless of scenario (72 vs 70.7), which is typical bandwidth-bound dense behavior — the entire delta comes from MTP acceptance. Same model, same draft head: code hits 7 out of 10, Japanese small talk barely clears a third.

Why Japanese chat misses so often

The payoff from speculative decoding is entirely determined by how well the draft predicts the real model's output. What's easy to predict is text with low next-token entropy.

• Code: syntax constraints are tight, and there's a lot of near-deterministic ground — boilerplate, indentation, keywords. The draft can harvest all of that.
• Japanese casual chat: sentence endings, phrasing, and interjections have genuinely high variance. At any natural point where "〜だね", "〜だよ", or "〜かな" are all equally valid, the draft has nothing solid to bet on.

On top of that, community reporting on MTP/speculative decoding skews heavily toward English + code benchmarks. If you carry a "2x faster with speculation" claim straight into a Japanese conversation product, expect to get less than half of that. Conversely, translation or template-heavy generation might still pay off in Japanese — that's unmeasured here.

Breaking down the "9x active-parameter gap, 2.5x measured speed gap"

Another number that defied intuition was the speed gap between MoE (3B active) and dense 27B. Parameter ratio is 9x, but the measured gap between 177 and 71 was only 2.5x.

Decode is bandwidth-bound, so what matters isn't parameter count — it's bytes of weight read per token:

• MoE 35B-A3B @ fp8: ~3B active x 1 byte + shared expert + attention ≈ ~4 GB/token
• dense 27B @ NVFP4: ~0.56 byte/param (+ bf16 embeddings etc.) ≈ ~20 GB/token

So the true ratio isn't 9x — it's ~5x (FP4 is half the bytes/param of fp8, which compresses 9x down to 5x). Comparing the theoretical ceiling to measured throughput narrows it further:

	Bandwidth theoretical (~1.7TB/s)	Measured	Ceiling utilization
MoE 35B-A3B	~425 tok/s	177	~42%
dense 27B FP4	~85 tok/s	71	~84%

Dense is running near its bandwidth ceiling, while MoE loses more than half its theoretical ceiling to kernel-launch and sampler overhead and similar fixed costs. The higher the raw tok/s, the more fixed costs compress the theoretical ratio — which is how 5x becomes 2.5x. "MoE is fast" is true, but on a single stream it's nowhere near the headline ratio — which also means dense's apparent disadvantage isn't as dire as the headline number suggests.

Operational conclusions

1. Don't design real-time Japanese conversation around speculative decoding paying off. At 36% acceptance, the gain isn't large enough to be perceptible. If conversation-path speed is a hard requirement, pick based on raw decode speed (= bytes/token)
2. For agentic/coding workloads, MTP is close to free. One flag, 1.85x. The head ships with the model, so extra VRAM cost is ~1GB
3. Measure acceptance on your own workload. A five-minute /metrics delta tells you everything. English-code benchmark numbers don't transfer to Japanese conversation
4. In the end we prioritized instruction-following and moved fully to dense 27B, accepting the 0.45x hit on the conversation path because we'd measured it first. What kept this from being a matter of gut feeling was having the per-scenario measurement before the decision, not after

---

Part of the record of rebuilding kotonia.ai's voice chat / desktop agent infrastructure. Related post: "When a local LLM lies about running Python" (the instruction-following side of this investigation).