What a 1.58-bit 27B Forgot — Measuring the Blind Spot Behind 'FP16 Quality' in Japanese

TL;DR

• PrismML released Bonsai 27B on 2026-07-14 — a ternary (1.71-bit) and binary (1-bit) conversion of Qwen3.6-27B, Apache 2.0. I ran it through a local eval the day after launch. The size and speed claims are real: on an RTX PRO 6000 Blackwell it hit 121.6 tok/s at 6.66 GiB of weights, beating the vendor's own H100 numbers. English conversation, code generation, and tool calling all held up at production quality
• Japanese, however, was broken. Across 50 prompts modeled on a voice-chat persona, 52% of replies leaked English words (a 7.3% Latin-character contamination rate). "三日坊主" (a Japanese idiom for giving up after three days) got translated literally into "three-day monk" with the whole reply switching to English; Simplified Chinese characters like 开 bled into supposedly Japanese sentences
• Running the same 50 prompts through the unquantized Qwen3.6-27B (Q4_K_XL) under identical conditions gave a 0.27% contamination rate and 6% tainted replies. The base model's Japanese is intact — the degradation was introduced during the ternary conversion, not inherited from the base
• The interesting part: the ternary (1.71-bit) and 1-bit variants degrade almost identically (52% vs. 56%). If this were a capacity problem, cutting bits further should make it visibly worse. It doesn't. What survives the conversion is decided by the training data distribution used during quantization-aware training, not by bit budget. That lines up cleanly with the fact that all 15 official benchmarks are English-only
• Bottom line: "94.6% of FP16" isn't a lie. It's 94.6% of what English benchmarks can see. Bring a model like this into a multilingual product unvetted and it can fail in ways no static benchmark will ever catch. Run a five-minute mechanical canary check (see below) before you adopt one

What Bonsai 27B is, and why it's a big deal

PrismML's announcement boils down to this: they took Qwen3.6-27B in its entirety and converted it to ternary ({-1, 0, +1}, an effective 1.71 bits/weight) and binary ({-1, +1}, 1.125 bits/weight). Weights shrink from 54 GB in FP16 to 5.9 GB (ternary) or 3.9 GB (binary). PrismML's own numbers: 11 tok/s on an iPhone 17 Pro Max, 44 tok/s on an M5 Max.

Sub-2-bit LLMs aren't new — the BitNet line has been doing this for a while. What makes Bonsai different is two things:

1. It converts an already-trained model instead of pretraining from scratch under a low-bit constraint. BitNet-style models require full pretraining under the bit constraint from the start, which meant throwing away every existing model asset (and capped BitNet models around the 2B range in practice). Bonsai reaches 27B by going "train a capable model first, compress it after"
2. There's no escape hatch. Conventional "2-bit quantization" typically leaves embeddings and attention in 4-8 bits, so the label and the reality diverge (PrismML's own whitepaper measures the so-called "2-bit" IQ2_XXS at an actual average of 2.8 bits/weight). Bonsai applies ternary/binary uniformly from embeddings through the LM head — the bit count on the label is the bit count you get

The whitepaper's headline claim: averaged across 15 benchmarks (all run in thinking mode), FP16 scores 85.07, ternary scores 80.49 (94.6%), and 1-bit scores 76.11 (89.5%). The same Qwen3.6-27B collapses to 72.73 under IQ2_XXS (9.4 GB) — Bonsai beats that by nearly 8 points at 60% of the size (5.9 GB). On AIME26, IQ2_XXS drops to 57.5 while ternary holds at 87.5. The method is a quantization-aware training (QAT) approach rooted in Caltech's Hassibi-lab line of research — ternary constraints are baked in from early in training, and the details are undisclosed IP.

I run a Japanese voice-roleplay platform (kotonia.ai) solo, and the production LLM is a finetune of the same Qwen3.6-27B family (ThinkingCap-27B NVFP4, roughly 20 GB) resident on an RTX PRO 6000 Blackwell. If the same 27B base can run in 7 GB, that reshapes how I allocate GPU memory. That was more than enough reason to test this the day after launch.

Test setup

• GPU: RTX PRO 6000 Blackwell Max-Q (96 GB). Voice services are isolated on a separate GPU, so production stayed up throughout testing
• Runtime: PrismML's llama.cpp fork (a custom quantization type, GGML_TYPE_Q2_0, plus CUDA kernels for hybrid attention — doesn't run on mainline llama.cpp), built from source for sm_120
• Four models compared on the same machine, same prompts:
  • Ternary Bonsai 27B (Q2_0, 7.2 GB)
  • 1-bit Bonsai 27B (Q1_0, 3.8 GB)
  • Unquantized Qwen3.6-27B (unsloth UD-Q4_K_XL, 17.6 GB) — the exact file the whitepaper uses as its FP16 stand-in
  • ThinkingCap-27B NVFP4 (vLLM, running in production) — for reference

First, the part that held up

I went in skeptical, but every infrastructure-level claim checked out under measurement. Some exceeded it.

Metric	Claimed	Measured (RTX PRO 6000 BW)
Weight size (ternary)	~5.9 GB (native target) / 7.2 GB (current GGUF)	6.66 GiB
Inference VRAM	8.4 GB at 4K context	8.9 GB at 16K context
Generation speed (tg128)	98 tok/s (H100)	121.6 tok/s
Prompt processing (pp512)	2,596 tok/s (H100)	3,210 tok/s

Decoding is bandwidth-bound, so a workstation GPU with better bandwidth-per-dollar than an H100 beating the vendor's number isn't surprising on paper. Still, seeing "27B in under 9 GB of VRAM at 121 tok/s" running locally lands harder than the numbers suggest. For comparison, the same environment runs unquantized Qwen3.6-27B Q4_K_XL at 65.7 tok/s. Weights shrink 2.46x and speed goes up 1.85x — the bandwidth math translates directly into the felt experience.

Quality held up too, as long as I stayed in English.

• English small talk and empathetic responses were natural — I couldn't find a breakdown
• Code generation passed both test cases I threw at it (interval merging, run-length compression), verified by actually executing the unit tests
• OpenAI-compatible tool calling returned correctly structured tool_calls
• Math: 10/10 correct on the same problem across 10 runs (though see the variance issue below)

The one thing that didn't match the claims was the bundled speculative decoder (DSpark, an EAGLE-style block-diffusion drafter). PrismML claims a 1.34x speedup on H100; in my environment it only moved the needle from 121.6 to 125.3 tok/s — a 3% gain. The draft acceptance rate for English essays was 73%, so the drafter itself works — it's that the base model is already fast enough that the drafter's forward-pass cost eats most of the win. And for Japanese, the acceptance rate drops to 10%. The drafter is English-trained too. This was the first sign something was off.

Kotone started speaking English

My product's voice persona uses short, conversational Japanese — 2-3 sentences, thinking mode off. Feed the same style of casual Japanese small talk to Bonsai and things go sideways fast.

User: 資格の勉強始めたけど三日坊主になりそう。("I started studying for a certification, but I think I'll give up after three days.") Ternary Bonsai: Qualification study began but three-day monk?

"三日坊主" (literally "three-day monk," an idiom for quitting almost immediately) got translated word-for-word, and the whole reply flipped to English. More examples:

User: 朝起きられないんだけど、何かいい方法ない?("I can't wake up in the morning, any good tips?") Ternary Bonsai: 朝起きられないのは、身体が休んで欲しいから maybe? 夜前にPhoneを遠くに置き、 morning light を受けられるようにカーテンを开けると、目覚め方が良いよ。

Along with English words bleeding in (maybe, morning light), "開ける" (to open, in Japanese kanji) came out as the Simplified Chinese character "开ける" — not just English contamination, but Chinese orthography leaking through as well. The 1-bit variant was even more blatant: it responded to a Japanese venting session entirely in English ("Yeah, totally get that. Sometimes boss behavior can be really frustrating.") and, when asked for karaoke song recommendations, suggested two songs that don't exist — "Hello Sunshine" and "Sugoi Nana."

Rather than leave this as anecdote, I measured it.

Quantifying it: Latin-character contamination rate

Using an LLM-as-judge invites arguments about reproducibility, so I went with something more mechanical.

• Send 50 casual Japanese prompts (work, relationships, food, health, hobbies...) with the voice persona's system prompt attached
• Strip whitespace from the reply and measure the share of characters that are Latin ([A-Za-z]), plus the share of replies containing a run of 2+ consecutive Latin characters

In natural spoken Japanese, loanwords render in katakana, so this number should sit near zero. That's exactly what the control models showed.

Model	bits/weight	Latin contamination rate	Replies containing English
Unquantized Qwen3.6-27B (Q4_K_XL)	5.2	0.27%	6%
ThinkingCap-27B (NVFP4, production)	~4	0.56%	2%
Ternary Bonsai 27B	1.71	7.29%	52%
1-bit Bonsai 27B	1.125	7.83%	56%

More than half of replies containing stray English words is close to a total loss for a conversational product. The same pattern shows up in Chinese too (a smaller 10-prompt probe put contamination at 0% for the base model and production model, versus 4.1% for ternary and 5.3% for 1-bit). Everything that isn't English got broadly pruned away, and English got concentrated.

To be fair to PrismML, the whitepaper doesn't hide that all 15 benchmarks are English, and it's upfront about regressions in instruction following and agentic tasks (IFBench drops from 68.0 to 58.5, for example). It makes no claims about multilingual capability at all, so nothing here is a broken promise. What isn't measured doesn't show up. That's the whole story.

Isolating the cause: the conversion process is the culprit

The obvious counter-argument is "maybe Qwen3.6-27B's Japanese was already weak." So I ran the exact same GGUF the whitepaper itself uses as its FP16 reference point (unsloth UD-Q4_K_XL) as a control. As shown in the table above: 0.27% contamination, 6% tainted replies. The unquantized base model's Japanese is completely intact, which confirms that the degradation was introduced during Bonsai's QAT conversion, not inherited.

The second finding is that degradation is nearly saturated between ternary (1.71-bit) and 1-bit (1.125-bit). Despite a 1.5x difference in bit budget, the Japanese breaks almost identically in both. If this were purely an information-capacity problem, the 1-bit model should be measurably worse. It isn't.

Which means this isn't a capacity problem — it's a question of what the conversion process was trained to learn. QAT re-learns representations optimized for ternary space, and whatever data distribution flows through that process becomes the capability distribution the model retains. Feed it English and code, and English and code survive. Don't feed it Japanese small talk, and Japanese small talk doesn't survive. The announcement's claim of preserving "27B-class thinking, reasoning, and agentic capability" is accurate — but the domain that capability spans is bounded by whatever the conversion data and benchmarks actually covered.

I think this matters less as a criticism and more as a way of understanding what this technique actually is. Sub-2-bit conversion looks less like "compression" and more like "distillation into an English-shaped subspace." And that framing cuts both ways — it implies a path forward. If someone runs the same conversion with enough Japanese in the training mix, Japanese would likely survive too, since the failure mode isn't capacity-bound in the first place.

A side finding: thinking-length variance

The whitepaper's benchmarks are all measured in thinking mode. What matters in production isn't the average — it's the variance. Running the same math problem 10 times, every run reached the correct answer, but the thinking trace length ranged from 2,339 to 7,934 tokens (a 3.4x spread). In a separate session, the same problem burned through 12,000+ tokens of thinking without ever reaching an answer. I haven't confirmed whether this is specific to sub-2-bit models (the base model shows some thinking-length variance too), but if you're wiring thinking mode into anything with a latency SLA, a token budget and a hard timeout aren't optional.

A checklist for anyone running a multilingual LLM product

Practical takeaways for anyone in the same position — shipping a user-facing LLM product outside the English-speaking world.

1. An English benchmark average tells you nothing about your users' language. This isn't specific to Bonsai — it applies to any model that goes through distillation, quantization, or pruning. The training data distribution used in that conversion is usually never disclosed
2. A canary metric takes five minutes to write. Your product's persona prompt, 50 prompts, and a Latin-contamination rate (or the equivalent "departure from the target language's script" metric for your language). No judge LLM required — fully reproducible, and cheap enough to run in CI on every model update
3. Speculative decoding's acceptance rate is language-dependent too. The drafter is smaller than the base model, so training-data bias shows up even more sharply there. Don't extrapolate an English-measured speedup to multilingual traffic
4. If you're adopting thinking mode, design capacity and timeouts around p99 thinking length, not p50

Even so, this is the start of something real

It might sound contradictory after spending this whole piece on a broken result, but my honest takeaway after finishing the eval is: this direction is real.

• Ternary 27B (80.49) beats FP16 8B (79.3). Holding a 5.9 GB ternary 27B is already smarter than holding a 16.4 GB 8B model — that crossover has already happened
• Because it's conversion-based rather than trained-from-scratch, it can keep pace with the frontier. Where the BitNet approach demanded "rebuild from scratch under a low-bit constraint," this approach just needs "convert whatever the newest capable model is"
• Since the failure mode isn't capacity-bound, a multilingual version should be buildable. If someone bakes a Bonsai variant with enough Japanese in the conversion data, there's a real path to running a top-tier local model on the GPU I already own

My product is a Japanese-language, voice-first, real-time combination that this model can't currently serve, so production stays on ThinkingCap-27B for now. But the canary metric this eval produced is going straight into my model-selection pipeline going forward. Fighting over quality that never shows up on a benchmark is just what running a multilingual product looks like.

Reproduction steps

Everything here reproduces from public artifacts.

# 1. Build PrismML's llama.cpp fork (CUDA, arch 120 for Blackwell)
git clone https://github.com/PrismML-Eng/llama.cpp llama.cpp-bonsai
cd llama.cpp-bonsai
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-bench

# 2. Get the models (HF: prism-ml/Ternary-Bonsai-27B-gguf, etc.)
#    Ternary-Bonsai-27B-Q2_0.gguf (7.2GB) / Bonsai-27B-Q1_0.gguf (3.8GB)
#    Control: unsloth/Qwen3.6-27B-GGUF's UD-Q4_K_XL (17.6GB)

# 3. Launch (official recommended sampling)
./build/bin/llama-server -m Ternary-Bonsai-27B-Q2_0.gguf \
  -ngl 999 -fa on -c 16384 --jinja \
  --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0
# To disable thinking mode per-request: chat_template_kwargs {"enable_thinking": false}

# 4. Benchmark
./build/bin/llama-bench -m Ternary-Bonsai-27B-Q2_0.gguf -ngl 999 -fa 1

The contamination-rate measurement script (50 prompts plus aggregation, under 100 lines of Python) is deliberately simple — I'd recommend rewriting the definition for your own target language rather than reusing mine verbatim. The more closely the prompt set matches your product's actual persona, the more sensitive it is as a canary.

---

kotonia.ai is a Japanese-language voice roleplay platform built on multilingual TTS and a lip-synced avatar. Running the LLM, TTS, and avatar generation off a single local GPU makes evaluations like this one both a hobby and a business necessity.