TL;DR
You can't hand an LLM an entire long conversation, so you summarize the old parts and fold them away. That's one sentence to describe. In Kotonia's character chat, I rebuilt how it folds five times.
- Replies that overflowed the context window were cut off mid-sentence, with no error at all
- Dropping old turns right away broke the prompt cache, and the turn that was trying to fold the history was the one that failed
- A summary rewritten from scratch on every fold could answer questions about older events at a rate close to zero. I switched to the same shape as generational garbage collection: stack chapters, and occasionally fold them into an older generation
- The summarizer couldn't tell dialogue from narration, so a lying character's claims about itself became facts in its memory
- I confused characters with tokens, and only Chinese conversations were losing the end of their memory
Who this is for
People building AI characters or any app with long conversations who are dealing with "the character forgets things from long ago." This is less about the summarization prompt and more about where to put the summary, when to make it, and how to measure it.
Background: one conversation is longer than the model's window
Kotonia is a service for role-play conversations with characters, and the relationship continuing over time is the product. The longest conversation is past 5,000 messages.
The amount a model can read at once (its context window) is limited. Production today runs GLM-5.3-Flash locally on llama.cpp, with a 64k-token slot per conversation. The character setup, persona images, conversation history and the reply about to be written all have to fit in it.
And at the time, loading the history had no LIMIT. Every turn re-sent every row of the conversation.
Here is the finished design in one picture up front. Old messages are summarized into a "synopsis" and "chapters," and recent ones go through verbatim. The rest of this article is about why each piece ended up the way it is.
(Budgets below are counted in characters, because the backend has no tokenizer. Most of the conversations are in Japanese, where one token is roughly 1.2–1.6 characters.)
1. It overflowed, and nobody noticed
The previous model had a 131k window, and long histories never hit it. On 2026-08-31 I switched to the current model, and they started hitting it immediately.
The way it failed was bad. In llama-server with --mmproj loaded for image input, the feature that drops old tokens and keeps going when the window fills (context shift) is forcibly turned off. Once the window is touched, generation simply stops, at whatever token it's on.
In one conversation the input grew from 13,475 tokens to 32,767, and from then on every reply was cut off mid-sentence. The backend didn't check finish_reason anywhere, so there was no error and no log. I only noticed because the pass rate of the reply-format check fell from 153/153 the day before to 45/67.
"Just cut off the old turns" wasn't an option for this service. The value is that the relationship continues; cutting would throw away the day you first met. So I decided to summarize what overflows and fold it away.
2. The turn trying to fold the history was the one that failed
The first implementation was straightforward. When the history went over budget, that same turn cut the old part and replaced it with a summary.
That broke something else. llama-server reuses the part of the prompt whose prefix matches the previous turn instead of recomputing it (the prompt cache). Dropping the front of the history changes the prefix, so the reuse is lost entirely. Production logs looked like this:
slot get_availabl: selected slot by LCP similarity, f_sim_best = 0.280 ← normally 0.97
slot release: stop processing: n_tokens = 29071, truncated = 0 ← cut at exactly 30 s
Re-reading 29,000 tokens takes about 29 seconds on an idle server and 58 when busy. The timeout was 30 seconds. And what was making it busy was the summarization job running at the same time. The process meant to rescue the conversation was what knocked it over.
Three fixes:
Don't fold on the turn that crosses the trigger. The threshold where folding starts (48,000 characters, about 30k tokens) is around half the 64k window, so a few more turns can go through as-is while the summary is being written without overflowing. Folding happens once, on the turn after the summary lands, and that's the only turn that loses the prompt cache. For the case where summarization keeps failing, there's a separate ceiling (56,000 characters) past which it cuts anyway.
Summarize in the background, 30 seconds later. One summary takes about 30 seconds in practice. The conversation can't go silent for 30 seconds. The delay is so it doesn't compete for the GPU with the turn currently running.
Don't put the summary in the system prompt. The system prompt is shared across every conversation with the same character, and it's the base of the prompt cache. Adding a per-conversation summary there would split that base per conversation. Instead the summary is inserted at the head of the history as a synthetic exchange: "here is the story so far." That's why, in the overview figure, the memory sits right after the character setup.
One more bonus failure: I raised the backend timeout to 150 seconds, but the frontend was closing the connection at 40. Fixing one side alone changes nothing.
3. A summary rewritten every time forgets the oldest things first
The first summary was a rolling one. Each fold rewrote a single summary from "previous summary + newly overflowed part."
After running it for a while, I tracked what was inside. Of the words in the first version of the summary, 20–30% remained after 4–5 folds, and 5% after 18. On every rewrite the summarizer favors recent events and drops the early premises. What disappeared was the information the story rests on, like "who is whose mother-in-law."
Making it generational
At the developer's suggestion, I gave it the same shape as generational garbage collection.
- Chapters (young generation): each folded range gets its own chapter summary, and chapters are stacked one by one. A chapter, once written, is never rewritten
- Old generation: when the chapters total more than 6,000 characters, the older half is folded into a single synopsis (a major GC)
Rewriting, and therefore attrition, now only happens during a major GC.
As the memory grows, it eats budget. If you keep the same amount of verbatim history while the memory grows, the interval between folds shrinks, and you get more heavy turns that lose the prompt cache. So the headroom right after a fold is fixed, and the verbatim history gets shorter by however much the memory grew (but at least 16,000 characters stay verbatim, so the current scene doesn't thin out).
Measuring how it forgets
To avoid ending at "it feels better," I built an offline evaluation.
- Split a real long conversation into ranges at the same points production would
- For each range, have an LLM write 8 factual questions about what happened in it
- Give the model only the memory as of the end of the conversation, have it answer, and grade it
Lay the ranges out from oldest to newest, and how much it remembers of how long ago becomes a curve.
| Approach | Memory length (conv. 1 / 2) | Mean recall (conv. 1 / 2) |
|---|---|---|
| Rewritten every time | 761 / 845 chars | 0.11 / 0.21 |
| Rewritten, longer | 1,823 / 2,131 chars | 0.30 / 0.49 |
| Stacked chapters (generational) | 3,817 / 4,409 chars | 0.56 / 0.69 |
The rewritten summary remembered only the newest range, and almost nothing before it. Stacked chapters could answer about half the questions even for old ranges.
However, I can't say how much of that gap comes from the structure. Generational memory is longer. The arm I prepared as "a rolling summary of equal length" only reached a little over 40% of the generational length even when told to be long (and pushing harder made it swing to nearly 5,000 characters). What I can say is that even a longer rolling summary kept the same curve shape, falling off toward older ranges.
The other limits of the evaluation:
- Only 3 Japanese conversations, 2 of them from the same user with the same character
- The same LLM writes the questions, answers them, grades them, and writes the summaries
- It measures whether facts can be answered, not how the conversation feels
- Summaries vary between runs. Running the same conversation 3 times, memory length ranged from 4,785 to 7,016 characters
Even so, after seeing a curve stuck at zero, that was enough reason to change approaches. When running the evaluation, the test binary needs to be pinned. If cargo test rebuilds every time, touching the working tree during an evaluation changes what is being evaluated (I nearly stepped on this).
4. In its memory, a liar becomes an honest character
A week after going generational, I was re-reading chapters in production and noticed something.
44 of 57 chapters contained the phrase "as an established fact." Looking inside, characters' claims about themselves (age, background) were written down as facts. Role-play characters lie. Some keep secrets, some fake their past. Those lines, after passing through memory, had turned into "established settings."
There were two causes. One was the summarizer's input: the character's replies were passed as bare text joined with spaces. Kotonia's text mode receives replies as JSON and keeps narration, dialogue (with speaker), and unspoken inner thoughts separate. That distinction was being thrown away right before the summarizer. The other was the instruction, which asked it to write "established facts to be referenced later."
Here is how the input and instructions changed (the example is fictional).
The instruction now has it write in-story characters' claims as "who said what to whom," and not write them as fact unless narration or inner thoughts back them up. Where a lie or act is apparent, it writes "on the surface" and "in fact" side by side. The memory header also says "a character's statement is a record that they said it, not necessarily the truth."
Overcorrecting, and losing what the user said
The first fix split things into two buckets: "what happened in the narration" and "what characters claimed." In evaluation, recall on a casual, everyday-chat conversation dropped from 0.72 to 0.50.
Most of the important facts in that conversation were things the user said about themselves. Tomorrow's plans, things they like, friends' names. Those are neither narration nor a character's claim, so they fit neither bucket and were dropped. Only in-story characters' statements should be doubted; what the user says about themselves is treated as fact. I added that.
Recounting in production
I also added an "audit" to the evaluation. It checks chapters against their source range and counts claims written as facts and topics left unresolved even though they were resolved. On 3 Japanese conversations, claims written as facts went 17 → 6, and recall 0.68 → 0.73.
I recounted in production chapters too (2026-10-27).
| Before (66 chapters) | After (67 chapters) | |
|---|---|---|
| Contains "established fact" | 38 | 0 |
| Closes a topic with "Resolved:" | 0 | 33 |
| Attribution like "said / told / claimed" | 5 | 16 |
However, leaving resolved topics as unresolved did not get better at all: 10 → 11 in evaluation. Adding instructions alone didn't work.
5. Smaller pitfalls
Explaining a refusal from n=1
The summarizer sometimes refuses on sensitive conversations: "I can't create that record." The first time I saw one, I concluded "putting the instructions at the end of the prompt causes refusals" and wrote that in a code comment. When I actually counted, refusals were 2 in 18 (about 11%) and had nothing to do with where the instructions were. Reproducing the exact same prompt passed 3 times out of 3. It was just a low-probability draw.
So it's handled in code rather than in the prompt. Short output containing stock phrases is treated as a refusal and retried once (the actual refusal was 101 characters and the shortest successful summary 519, so the cutoff is 400). Never save a refusal as a summary. If you do, "I can't help with that" sits in the synopsis slot on every turn from then on.
Incidentally, the summarizer is given an in-story role, "the chronicler of the story," but not to reduce refusals. Asking plainly to "summarize" sometimes returned a continuation of the story instead of a summary.
Confusing characters and tokens
The Rust side has no tokenizer, so budgets are counted in characters, using a measured 1.57 characters per token. But that was the value for the input side, JSON included. For output summaries it's 1.2–1.3 for Japanese, 1.2 for Chinese and 4.7 for English characters per token, a 4x spread across languages.
The major GC's output cap was set as "target 1,500 characters + 400" = 1,900 tokens, so 5 of 11 Chinese synopses were cut off mid-sentence. A cut-off synopsis becomes the input to the next major GC. The end of the synopsis is where open threads are listed, so the unfinished storylines are the first to erode. The cap is now a ceiling that exists only so it doesn't cut (4,096 tokens).
Don't fold for models with wide windows
The 48,000 figure above is for the local model, and what sets it isn't so much the window size as the time to re-read when the prompt cache misses. Applying the same threshold to a cloud model with a 1M window means replacing conversation that could have stayed verbatim with a summary. DeepSeek / Gemini / Grok keep about 200,000 characters verbatim.
Conversely, the local summarizer can't read a large range in one go. When a conversation kept long verbatim on a cloud model switches to the local model, it would have to fold 150,000 characters at once. In that case the range is split every 32,000 characters and stacked as multiple chapters, oldest first.
6. Results in production, and what isn't fixed
Logging started on 2026-10-10 (as of 10-27).
| Count | Median time | Mean output length | |
|---|---|---|---|
| Writing a chapter | 133 | 27 s | 1,110 chars |
| Major GC | 17 | 34 s | 4,123 chars |
| Forced cut at the ceiling | 3 | — | — |
One fold folds about 17,000 characters on average. During the logged period, summaries that failed due to refusal: 0 (absorbed by the retry).
To be honest, not many conversations hit this yet. Of the roughly 40 conversations where compaction ran, 28 are the developer's own. It only kicks in for people who talk for a long time, so it matters more as users stick around.
What isn't fixed:
- Resolved topics stay listed as "unresolved"
- English and Chinese synopses ignore the target length and grow with every major GC
- Evaluation is Japanese only and doesn't measure how the conversation feels
- There's no append-only ledger of facts yet, and no mechanism to keep signature lines verbatim
Closing
What worked, before polishing the summarization prompt, was a set of placement decisions: where to put the summary (somewhere it doesn't split the cache), when to make it (outside the turn, delayed), and what never to rewrite (chapters only get stacked). And what let me change those decisions was seeing forgetting as a curve.
Once you've seen a line stuck at zero, you can't say "it's summarized, so it's fine" anymore.