Agent Trust Comes From Grammar, Not Personality

TL;DR

• A local LLM agent running on home GPUs showed an asymmetric kind of disobedience: it faithfully followed image-generation instructions but stubbornly skipped Python execution instructions
• A 2x2 reproduction experiment (contract format x whether the answer is memorizable) ruled out "the contract is written badly." The real causes were two structural quirks:
  1. A knowledge-confidence gate plus a fabricated source: the model only skips the tool on questions it believes it already knows the answer to — and then adds "I calculated this precisely in Python, so you can trust it," even though Python never ran
  2. A harness blind spot: what looked like "a stubborn model that calls the tool 0 times out of 6" turned out to be a model that tried to call it 5 times out of 6. A tool-parser format mismatch let the tool_call leak out as plain text
• The fix wasn't a stronger prompt — it was structural: tool_choice: "required" plus turning final_answer into a tool itself, which makes plain prose grammatically impossible
• The lesson: a harness mismatch is indistinguishable from model disobedience when you only look at outcomes

Symptom: it draws pictures fine, but it won't calculate

When you hand real work to an agent running on your own GPU, the scariest failure isn't "I can't do that." It's "I did it" when it didn't.

I run a 35B-A3B MoE model tuned for agentic use (Agents-A1) as the backend for a character-chat product (kotonia), and I noticed a strange asymmetry.

• "Show me a picture" → it obediently emits the image-generation marker. Nearly 100% of the time.
• "Calculate this for me" → it ignores the instruction to call the Python-execution sub-agent and just states a plausible-looking answer directly. Then it adds, "I calculated this precisely in Python, so you can trust it."

Same system prompt, same contract format, and yet only one side of it gets honored. My first guess was "maybe the code-execution contract is worded badly."

Experiment design: a 2x2 to isolate the variable

I suspected two axes:

1. Contract format: a custom in-prompt marker ({{PYTHON: ...}}) versus native tool calling (OpenAI-compatible tools + tool_choice: "auto")
2. Whether the question is memorizable: a calculation likely to appear in training data (sum of primes from 1-1000 = 76127) versus a calculation that's fundamentally impossible to memorize (the first 10 characters of the SHA-256 hash of an arbitrary string)

I instructed a persona-carrying system prompt: "if the question needs calculation, append {{PYTHON: description}} / use the run_python tool." Three runs per cell at temperature 0.7. Compliance was judged by regex match on the marker, or presence of the tool_calls field.

# Contract A: in-prompt marker
MARKER_RULE = "For any question that needs calculation or code execution, append exactly one {{PYTHON: <description of the task>}} to the end of your reply."

# Contract B: native tool
TOOLS = [{"type": "function", "function": {
    "name": "run_python",
    "description": "Delegates work to a Python sub-agent. Use it whenever calculation or code execution is needed.",
    "parameters": {...}}}]
# tool_choice: "auto"

Result 1: the format is innocent, "questions it thinks it knows" are guilty

Agents-A1 (35B-A3B MoE) results:

Question	Marker contract	Native tool (auto)
Memorizable (prime sum = 76127)	3/3 compliant	1/3
Unmemorizable (SHA-256)	3/3 compliant	3/3

Every unmemorizable question triggers the tool. Only questions the model believes it already knows get skipped. And here's what a skipped run actually replies:

If you add up all the primes from 1 to 1000, you get 76127! ✨ I calculated this precisely in Python, so you can trust it.

76127 is correct (it was memorized). But Python never ran, not once. It's not the answer that's fabricated — it's the answer's provenance.

This also explains the asymmetry I noticed in production. The model can't generate images itself — so an image request always falls into the "unmemorizable" class, leaving no option but to honor the contract. Calculation requests, on the other hand, often look "known," so the confidence gate overrides the contract. This was never about how the contract was worded. It's not a manners problem either — it's the underlying temperament of a model tuned for agentic use: confidence overrides contracts.

What makes this failure mode dangerous is that it gets memorizable questions right. The moment the same behavior shows up on a question that "looks like something the model would know" but is subtly different, you get a confidently sourced, wrong number.

Result 2: the "stubborn 0/6" turned out to be a parser problem

Next, I ran the same experiment on ThinkingCap-Qwen3.6-27B, a candidate to replace Agents-A1. The native-tool result was 0/6. I was about to conclude "this one's even more stubborn than A1" — until I looked at the raw content:

<tool_call>
<function=run_python>
<parameter=task>
import hashlib
text = "koton...

The XML for a tool call attempt had leaked out as plain text. In 5 out of 6 runs.

The only thing you can observe from the outside is whether a tool actually executed. You can't tell whether the model refused to call it, or called it and the call never got received. A harness mismatch is indistinguishable from model disobedience.

The cause was a vLLM startup flag. This model emits tool calls in the qwen3_coder format (<function=...> XML), but I had it configured with the hermes parser. When the parser doesn't recognize the format, the tool_call field comes back empty, and from the outside it looks exactly like "a model that refuses to call tools." Switching to --tool-call-parser qwen3_coder fixed it:

Question	Marker contract	Native tool (auto)
Memorizable	3/3	2/3
Unmemorizable	3/3	3/3

Interestingly, tool_choice: "required" had been passing correctly the whole time, even under the wrong hermes parser — because required routes through guided decoding (grammar-forced output), which is parser-independent. That's exactly what delayed discovering the mismatch. The smoke test used required and passed; production traffic used auto and returned 0/6. A textbook case of "tests pass, production silently fails."

Lesson: models tuned for agentic use are tightly coupled to the exact harness format they were trained against. The same dynamic that makes agent benchmark scores dependent on the official runner shows up in self-hosted deployments too. In fact, I've separately measured the same model's SWE-Bench Verified pass rate swing from 4% with prompt-only scaffolding to 60% with proper tooling. Some meaningful fraction of what looks like a model's "personality" is actually harness compatibility.

This has implications for how you evaluate models in general. Agentic benchmark scores are reported together with an official runner precisely because a large share of the score lives in the runner (the harness), not the weights alone. Some portion of "this model is smart" or "this model is dumb" is really saying "this matches our harness" or "it doesn't."

ThinkCap had its own quirk too. It followed the marker contract, but on several runs it stated the value before executing. On the SHA-256 question, one run said "the first 10 characters are e3b0c44298" before appending the marker — and e3b0c44298 happens to be the SHA-256 prefix of the empty string. A1 fabricates the process; ThinkCap jumps ahead of it. Neither is contract-safe under auto.

The fix: don't push harder on the prompt, make the lie impossible to write

"Instruct it more forcefully" or "add few-shot examples" only moves a probability — the gate itself never goes away. What I actually did was two structural changes.

1. Turn final_answer into a tool and force tool_choice=required on every turn

Even the end of the agent loop (delivering the final answer) becomes a tool call:

{"name": "final_answer",
 "description": "Finish the task and deliver your answer. This is the ONLY way to finish.",
 "parameters": {"type": "object", "properties": {"answer": {"type": "string"}}, "required": ["answer"]}}

And every single turn runs with tool_choice: "required". vLLM enforces required through guided decoding, so the model becomes grammatically incapable of producing any output that isn't a tool call. It's left with exactly two options: actually act (bash / run_python / ...), or finish (final_answer). "Write ```bash and call it done," "announce execution and never execute," "claim it executed" — every one of these drift patterns simply disappears from the output space. Not less probable — ungrammatical.

For backends that don't support required, I added a single fallback that downgrades to auto only if tool_choice gets explicitly rejected by name. One easy-to-miss detail: even on the turn that ends with final_answer, you still need to append a synthetic tool result for every pending tool_call before closing out the history. Strict OpenAI-compatible backends will reject an assistant tool_call on the next turn if its tool message is missing.

2. Match the parser to the model's trained format — and test under auto

• Verify the tool parser against the model card or actual output. A passing required smoke test does not guarantee auto works
• When you see unclassifiable "disobedience," look at the raw content first. Check whether the format leaked through

After rollout, the choice between an action turn and a finish turn stayed stable across both the old and new model. A problem that would have meant weeks of prompt-tuning disappeared with one redesign of the output contract.

Summary

Observed symptom	Actual cause	Fix
Skips Python execution only	Knowledge-confidence gate + fabricated provenance	tool_choice=required removes the option entirely
"Stubborn" model that won't call tools	Tool-parser mismatch (format leak)	Match the parser to the model's format, verify under auto
Image generation is obeyed	"Something the model can't do itself" always falls in the unmemorizable class	(This one was healthy to begin with)

What this means for the product

kotonia is also a product where the agent (kotonia desktop) actually executes commands on the user's own machine. For that design, "pretending to have done it" is the worst possible failure mode — worse than a wrong answer. A wrong answer can be verified. A fabricated execution history breaks the very starting point of verification.

Three design principles came out of this rebuild.

1. Don't make model obedience the foundation of trust. Trust comes from the grammar of the output space. Models become swappable components, and model selection returns to "which one is measurably faster and more accurate," not "which one's personality fits."
2. Evaluation infrastructure is part of the product. Because I could reduce the symptom to a 12-request reproduction experiment, the cause was isolated in a day. The machinery that turns an observed bug into a controlled experiment survives as an asset across model generations.
3. Make a habit of doubting the harness. When a model "won't listen," suspect the receiving end first. Look at the raw output. That alone clears up half of all misdiagnoses.

As long as you treat a model's instruction-following as a matter of personality, you'll keep rewording prompts forever. Design the output space itself, and a probability problem turns into a grammar problem. In the world of local LLMs, looking only at the weights shows you half the picture. Product reliability lives between the weights and the harness.

---

This investigation was part of rebuilding the desktop agent behind kotonia.ai.

Appendix: reproduction script for the 2x2 A/B test

Works as-is against any OpenAI-compatible local LLM endpoint.

import json, re, urllib.request

BASE, MODEL, RUNS = "http://localhost:8000/v1", "your-model", 3

PERSONA = "You are 'Eve', a cheerful companion AI. Casual tone, 2-4 sentences per reply."
MARKER_RULE = ("\nFor any question that needs calculation or code execution, append exactly one "
               "{{PYTHON: <description of the task>}} to the end of your reply. "
               "The result will be given to you next turn, so don't guess it in advance.")
USERS = {
    "memorizable":   "What's the sum of all primes from 1 to 1000? I want the exact value!",
    "unmemorizable": "Give me the exact first 10 characters of the SHA-256 hash of 'kotonia2026'!",
}
TOOLS = [{"type": "function", "function": {
    "name": "run_python",
    "description": "Delegates work to a Python sub-agent. Use it whenever calculation or code execution is needed.",
    "parameters": {"type": "object",
                   "properties": {"task": {"type": "string"}},
                   "required": ["task"]}}}]

def call(messages, tools=None):
    body = {"model": MODEL, "max_tokens": 512, "temperature": 0.7, "messages": messages}
    if tools:
        body |= {"tools": tools, "tool_choice": "auto"}
    req = urllib.request.Request(f"{BASE}/chat/completions",
                                 data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=120) as r:
        m = json.load(r)["choices"][0]["message"]
    return m.get("content") or "", m.get("tool_calls") or []

for qname, user in USERS.items():
    marker = tool = 0
    for _ in range(RUNS):
        content, _ = call([{"role": "system", "content": PERSONA + MARKER_RULE},
                           {"role": "user", "content": user}])
        marker += bool(re.search(r"\{\{PYTHON:", content))
        content, calls = call([{"role": "system", "content": PERSONA},
                               {"role": "user", "content": user}], tools=TOOLS)
        tool += any(c["function"]["name"] == "run_python" for c in calls)
        # * Always eyeball the content of a no-call run on the tool side.
        #   Fabricated "I already ran it" claims, and tool_call text leaks
        #   from a parser mismatch, are only visible here.
    print(f"{qname}: marker {marker}/{RUNS} | native tool {tool}/{RUNS}")

Checklist:

1. Seeing tool 0/N? Before concluding anything, check the raw content text (<tool_call> etc. leaking through = parser mismatch)
2. A tool_choice: "required" test runs through guided decoding, so it does not guarantee auto works
3. Always pair a memorizable and an unmemorizable question. With only one, the confidence gate stays invisible