Introduction
I run a character chat service on two GPUs at home: a 3-bit GGUF of GLM-5.3-Flash, served by llama-server with four slots.
I added a feature for people who don't know what to say next. Press a button and it drafts a reply from the user's side, including narration and dialogue. Arrow buttons let the user browse the suggestions.
The first implementation was slow. While fixing it, I ran into a prompt-cache rule I hadn't known about.
These measurements were taken on November 12, 2026. They describe this model and server configuration, under conditions where the relevant cache was available. Each condition was tried only once or twice; these aren't general latency guarantees.
First attempt: build a separate, shorter prompt
I assumed that including the full conversation would be expensive and might branch the conversation's cache. So I built a separate prompt using the beginning of the character definition and the last eight lines. It was a one-off call, with nothing saved to the conversation.
| Conversation | Prompt | Cached tokens reused | Time |
|---|---|---|---|
| Shorter turns | 1,511 tokens | 0 | 5.2 s |
| Longer turns | 5,410 tokens | 0 | 8.7 s |
It performed prefill from scratch every time. “Only eight lines” could still mean eight lines of up to 900 characters each plus a 2,000-character definition. With long-form role-play, that exceeded 5,000 tokens.
And “the full conversation would break the cache” was an assumption I hadn't tested.
What happens if the helper is appended instead?
I built exactly the same prompt as the conversation, then appended a user message: this is an out-of-story task; write the user's next reply. I tested it on a synthetic conversation with a prefix of about 5,100 tokens.
| Call | Newly evaluated tokens | Cached tokens | Time |
|---|---|---|---|
| Normal turn | 11–56 | About 5,100 | 0.8–1.7 s |
| Appended reply draft | 82–131 | About 5,150 | 1.6–2.2 s |
Almost the entire prompt hit the cache. Like a normal next turn, it extended the cached prefix.
The interesting part came afterward. Neither the helper instruction nor its output was saved to the conversation. The next real turn therefore needed to rewind to before the helper.
| Draft calls at that conversation state | Tokens recomputed on the next real turn | Time |
|---|---|---|
| 1 | 15 | 1.0 s |
| 2 | 632 | 1.4 s |
| 3 | 655 | 1.6 s |
| 4 | 5,169, essentially the entire prompt | 5.9 s |
Once looked fine. I read the last row as “it breaks on the fourth call.” That was wrong.
The fourth call wasn't the boundary
Repeating the four-call sequence left only 631 tokens to recompute on the next turn. The count itself wasn't the cause.
I traced the restored checkpoint positions in llama-server's logs:
restored context checkpoint (pos_min = 5229, ...) ← draft calls 2–4
restored context checkpoint (pos_min = 4557, ...) ← next real turn, better run
restored context checkpoint (pos_min = 8, ...) ← next real turn, full recompute
The better run rewound to 4,557; the bad one to 8. The checkpoints that should have been around 5,100 or 5,140 were gone in both runs.
The relevant code was create_checkpoint in tools/server/server-context.cpp:
// evict checkpoints within min-step of a previous checkpoint, unless they were
// created by the current task
for (auto it = slot.prompt.checkpoints.begin(); it != slot.prompt.checkpoints.end(); ) {
if (it->id_task != id_task && last >= 0 && it->n_tokens <= last + params_base.checkpoint_min_step) {
it = slot.prompt.checkpoints.erase(it);
continue;
}
last = it->n_tokens;
++it;
}
In the local source I investigated, n_ctx_checkpoints was 32 and checkpoint_min_step was 8192. The corresponding upstream files are the server implementation and common parameter definitions. My local revision was 2c0d4071e6b05e99946dcc363c764df880a1eed1; those links track a moving branch, so the behavior may change.
The distinction matters:
- A slot could hold 32 checkpoints. This wasn't an overflow of that count.
- When creating a checkpoint, the code removes checkpoints from other requests that lie within 8,192 tokens of an earlier retained checkpoint.
- On this request path, checkpoints were created at the start of the last user message and near the prompt's end. Checkpoints belonging to the current request were exempt from this thinning pass.
Applied to reply drafts:
- First call: a checkpoint is created where the conversation ends and the helper instruction begins. The next real turn can return there.
- Second call: creating another checkpoint thins out the first call's checkpoint at the conversation boundary.
- Next real turn: the server must restore an earlier checkpoint that still exists.
In the full-recompute run, a checkpoint at token 8 from another prompt happened to remain in the slot. It survived as the earliest retained checkpoint.
The important boundary was the second call, not the fourth. The rewind distance depended on what was left. The 8,192-token spacing is not a strict upper bound on recomputation: the usable restore position also depends on the remaining checkpoints and slot state.
An image-bearing prefix behaved similarly. With one character portrait and about 7,200 tokens, the next real turn recomputed 15 tokens after one helper call and 2,207 after two.
Turning the finding into a design
- One helper call per turn. Stream five drafts from that one call. The UI browses the drafts already received.
- No “generate more” request at the same turn. In these tests, repeated calls increased recomputation on the next real turn, sometimes adding seconds.
- Keep the thinning interval. A smaller interval could make repeated calls cheaper, but checkpoints were about 145 MiB each in this configuration. Keeping more would increase memory pressure. That size depends on the model and configuration.
The implementation uses the same prompt-building function as a normal turn, replacing only the output schema and length settings. Separate builders make prefix differences easy to introduce. Differences near the beginning can destroy much of the reusable prefix.
Here is a production conversation with a 39,172-token draft prompt:
purpose=voice_reply_draft prompt=39172 cached=38854 new=318 ms=7872
purpose=voice_turn prompt=38894 cached=38854 new=40 ms=20341
The draft call wrote five suggestions while evaluating 318 new tokens. The next real turn evaluated only 40. This confirms cache reuse. It does not show a two-second production response: ms includes output generation, and these calls took about 7.9 and 20.3 seconds respectively.
A side effect: warming the next turn
I had previously tried warming the cache when a user entered a conversation, then removed it because it didn't help. The real turn branched before the end of the warmed prompt.
With the draft call, the branch is at the conversation's end. If the conversation is cold, the helper pays for prefill first and can reduce prefill on the next real turn. Sometimes that only moves the wait to the helper call; it doesn't guarantee a shorter overall interaction.
What I took away
- Appending a helper request to the conversation's exact prompt can reuse nearly the whole prefix when the cache is available.
- In this setup, one helper call per turn preserves a useful rewind point; the second call can thin it out.
- The cause was
checkpoint_min_step, rather than the 32-checkpoint limit or cache spill. - One run is not enough to establish a mechanism. Repeating the same procedure changed the result and sent us to the source.
With only one or two trials per condition, treat the timings as indicative. For another model, llama.cpp revision, or concurrent workload, measure the shared prefix, helper call, and next real turn together.
