Introduction
I run a character chat service on a local LLM with two GPUs at home. The model is a 3-bit GGUF of GLM-5.3-Flash, served by llama-server with four slots.
One day I looked into something that had been bothering me: while one particular user was talking, inference got very slow for everyone.
The cause wasn't how that user behaved, the GPUs, or the cache size. llama-server was correctly restoring a cache entry from RAM, then discarding it a few lines later.
Miss rate on the path that restores from the RAM cache (3,000+ tokens)
Without LoRA 42% (23 of 55)
With LoRA 77% (61 of 79)
After the fix, the same count put the miss rate with LoRA at 15–22%. An upstream PR with the same approach already existed. I suggested a regression test that reproduces the production symptom in a comment, and it was added to that PR. The PR is still awaiting review and had not been merged as of 2026-11-07.
This is a record of that investigation, including the hypotheses I got wrong.
The symptom
That user only used text, with no voice or image generation. Their prompts had a median of 32K tokens: someone carrying on a long conversation.
Over 16 hours of logs, 25 of their 40 turns missed the cache, and each miss triggered 40–68 seconds of prefill. The effect on other users looked like this.
| Situation (last 3 days, generation time per character) | Count | Median | p90 |
|---|---|---|---|
| Neither is running | 1,025 | 24.5 ms | 66 ms |
| That user is generating | 29 | 61.3 ms | 204 ms |
The sample is small, and long requests are more likely to overlap, so the ratio is rough. Even so, generation dropped from 58 tok/s to 19 tok/s during periods with overlapping prefills.
How a prefill stops other people's generation is a separate problem, covered in another article. The question here is: why were 30K tokens being re-read every time in the first place?
Hypothesis 1: the cache is too small
The llama-server log was full of lines about evicting old entries.
alloc: - making room for prompt cache entry, removing oldest entry (size = 747.843 MiB)
The RAM prompt cache was at its limit. That looked like the cause.
Here came the first mistake. While investigating with Claude, we read this as "--cache-ram isn't set, so the limit is the default 8 GiB", and Claude recommended raising it to 16 GiB. It was already 16 GiB. We had built an explanation from a default value without looking at the real one.
More importantly, the capacity theory fell apart once we counted. I split 639 turns of 3,000+ tokens by how their slot was selected.
| How the slot was selected | Hit | Miss |
|---|---|---|
| LCP (the conversation was still in a GPU slot) | 381 | 10 |
| LRU (the slot was in use elsewhere; restore from RAM) | 107 | 114 |
Almost all misses were on the RAM path. But 58 turns missed even though they came less than 5 minutes after the previous turn. With entries of 600–1,000 MiB and a median eviction interval of 5.3 minutes, a conversation should stay in the cache for 50 minutes or more. It couldn't vanish in 5. The miss rate was also the same, about 33%, whether or not an eviction happened between turns.
More capacity wouldn't fix those 58 turns.
Hypothesis 2: regeneration changes the history
This was my guess. Regenerating or deleting a message forks the history, so the cache shouldn't match.
It was half right. Regeneration missed 8 times out of 24. But that was a small share of the total. The most common case was a conversation simply continuing, with the prompt just getting longer, and still missing.
No logs, and TRACE isn't an option
I wanted to see the decisions, but lines such as "which cache candidates were compared" and "was this restored from a checkpoint" were at TRACE level.
There's a trap here. Enabling TRACE also enables DEBUG, which writes every prompt token to the log. On a production service, I can't send users' conversations to the journal.
So in our fork, I raised just the nine lines that output positions and sizes to INFO.
- SRV_TRC(" - looking for better prompt, base f_keep = %.3f, f_sim = %.3f\n", ...);
+ SRV_INF(" - looking for better prompt, base f_keep = %.3f, f_sim = %.3f\n", ...);
A strange example appeared right away. One turn found and restored an entry with a 97% match (37,906 of 39,027 tokens), then prefilled all 39K tokens from the start, taking 68 seconds. There was no log of a checkpoint restore, and no log of a full re-process either.
The state was restored. It just wasn't used.
The cause: it discards the state right after restoring it
LoRA looked suspicious, so I split the numbers by whether one was in use. At the time, this service specified lora per request and applied it only to the turns of certain characters.
| Stayed in the same slot | Restored from RAM | |
|---|---|---|
| Without LoRA | 3% miss | 42% miss |
| With LoRA | 4% miss | 77% miss (61 of 79) |
Only the bottom-right cell stands out. Reading the code showed this sequence:
- LoRA settings are stored per slot, not in the cache entry
- A conversation with a LoRA gets evicted, and someone else uses that slot without one
- The original conversation comes back, and its state is restored correctly from RAM
- But the slot still holds the previous user's setting (no LoRA)
launch_slot_with_taskdecides "the LoRA changed" and clears the state it just restored
if (!are_lora_equal(task_loras, slot.lora)) {
if (lora_should_clear_cache(slot.lora, task_loras)) {
SLT_TRC(slot, "clearing cache for lora change..."); // TRACE, so invisible
slot.prompt.clear();
}
...
}
The state for a LoRA was restored for a request with that same LoRA, then discarded after being compared with an unrelated previous user's value. The log line is TRACE, so nothing shows up.
Doubting the numbers once more
The 42% and 77% pair bothered me, and I asked whether it was a matter of ratios. Maybe turns that could never hit were mixed in.
A recount showed exactly that.
| First LLM turn | Right after summarization | 100+ min gap | Conversation continues | |
|---|---|---|---|---|
| Without LoRA | 10 of 10 | — | 2 of 2 | 10 of 37 (27%) |
| With LoRA | 4 of 4 | 3 of 3 | 4 of 4 | 35 of 52 (67%) |
The 42% without LoRA was heavily inflated by turns that can't hit in principle. Comparing only continuing conversations, which should hit, gives 27% and 67%. That gap of about 40 points is a much clearer picture of the LoRA effect.
The fix
The approach is simple: store which LoRA configuration a state was computed with in the cache entry.
- Keep the LoRA configuration in each entry
- When restoring, only consider entries whose configuration matches the request
- After restoring, update the slot's configuration to match the entry
- Don't reuse a state computed with a LoRA for a request without one
Which LoRA to apply is still decided per request, as before.
A synthetic request reused 6,023 of 6,036 tokens. In production, the RAM path reused 7,137 tokens with a 1.5-second prefill and 8,635 tokens with a 1.7-second prefill.
Also found: eviction by health checks
The new logs showed something else odd. A tiny 21-token request arrived about every 6.5 minutes and took over the slot of a big conversation.
The culprit was my own health check. Two causes overlapped:
- The start of the chat template alone gives a 42% match, above the 10% threshold, so a big conversation's slot counts as "similar"
- With
--kv-unifiedon this version, every task start saved all idle slots to RAM and cleared them
I first considered a dedicated path for health checks, or detecting one-off requests, but no detection was needed. A short prompt uses almost no KV cells. In the fork only, tasks under 256 tokens no longer evict idle conversations or take a big conversation's slot. After this, conversations stayed in their slots even right after a health check, with no RAM restore needed at all.
This part of upstream has since been rewritten, so the fix probably isn't needed there. I haven't confirmed that.
Results in production
I repeated the same count on logs from after the fix. I also re-ran the count on the old logs and got the same "before" values.
| Miss rate restoring from RAM | Before (16 hours) | After (Nov 1–2, about 25 hours) | Later (Nov 2–6) |
|---|---|---|---|
| Without LoRA | 42% (23/55) | 36% (18/50) | 21% (81/383) |
| With LoRA | 77% (61/79) | 22% (17/77) | 15% (68/459) |
For continuing conversations only, misses after the fix (evening of Nov 1 to Nov 6) were 3.5% without LoRA (33/932) and 2.9% with LoRA (35/1,197). The difference between them is gone.
Most remaining misses can't be avoided by design.
| Misses after the fix (3,000+ tokens) | Without LoRA | With LoRA |
|---|---|---|
| First turn of a session | 74/80 | 32/40 |
| Right after history summarization | 10/10 | 24/28 |
| 100+ min gap | 14/20 | 13/20 |
The summary is inserted at the start of the history, so everything after it changes. For conversations around 30K tokens, the full prefill there takes about a minute. That is still an open problem.
This comparison has several caveats:
- The before/after numbers for continuing conversations were counted differently. Before the fix, they are estimates from matching timestamps with the database. After it, they come from logs tagged with a session ID
- The LoRA fix and the health check fix went in on the same day, so their effects can't be separated. Removing health check evictions also takes easy-to-hit turns off the RAM path, which changes what its miss rate means
- On Nov 2, I raised the RAM cache limit to 24 GiB and the shared KV to 256K. The rightmost column includes that effect
- The mix of users varies by day. I didn't re-measure under the same load as the original 16 hours
- The count stops in the early hours of Nov 6. After that, I moved almost all turns to one LoRA setting, so later data can't be used for this comparison
I went to report it upstream, and a PR already existed
Upstream had the same code, so I started preparing a PR. But PR #30275, with the same approach, had been open since October 10.
Its motivation was the correctness problem in #30129, where KV from different LoRAs gets mixed. What I saw was a performance problem: KV for the same LoRA being discarded every time. The root is the same, but the symptoms point in opposite directions. The PR's test checked the former.
I also checked llama.cpp's contribution policy:
- Don't open duplicate PRs; comment on the existing one
- AI-written code is allowed if disclosed
- AI must not write the text of issues, PRs, comments, or commit messages
So Claude handled the reproduction environment, the test code, and organizing the numbers, while I wrote the comment myself. When Claude reviewed my English, it pointed out factual errors such as "miss rate, not hit rate" and "72% is a typo for 77%". It also pointed out that I had only verified my fork in production, not the PR itself, so I shouldn't write that this fixes everything.
The test is just this: use one slot, and send requests with a LoRA, without one, then with one again.
def test_lora_ram_prompt_cache_restore_same_lora():
global server
server.n_slots = 1
server.n_ctx = 2048
server.n_predict = 4
server.cache_ram = 256
server.start()
prompt = "Look in thy glass and tell the face thou viewest. " * 15
other = "The quick brown fox jumps over the lazy dog. " * 15
lora = [{"id": 0, "scale": 0.5}]
def complete(text: str, lora_cfg):
data = {"prompt": text, "temperature": 0.0, "cache_prompt": True}
if lora_cfg is not None:
data["lora"] = lora_cfg
res = server.make_request("POST", "/completion", data=data)
assert res.status_code == 200
return res.body
first = complete(prompt, lora)
complete(other, None) # evicts the first conversation from the only slot into RAM
res = complete(prompt + "And then", lora)
n_prompt = first["timings"]["prompt_n"]
assert res["timings"]["cache_n"] >= n_prompt - 1
It only looks at cache_n on the third request. It needs neither the production GPU nor the production model, and reproduces on a CPU with a tiny test model.
| This test | |
|---|---|
| Upstream (060b2fe) | Fails (cache_n = 0, prompt_n = 183) |
| With PR #30275 applied | Passes |
A few days after I posted, the PR author replied: they had added a regression test based on it in 4e94dc5. In their environment it also showed a difference, with 181 cached tokens on the PR and 0 on the baseline (1623d8c).
What I learned
1. Test a plausible cause against how the misses actually happen. The eviction logs were real, and the capacity theory made sense. The 58 misses within 5 minutes were what it couldn't explain.
2. When you see a percentage, ask what's in the denominator. The 42% versus 77% comparison included turns that could never hit. The 27% and 67% from splitting them out describe the bug's effect much better.
3. Code that silently discards work has no logs. The decision lines were TRACE only, and TRACE also outputs user text. Performance bugs don't throw exceptions, so making them visible has to come first.
4. Record measurements so you don't have to join them later. Matching server logs and the database by timestamp has limits. Now each call writes one line with its purpose, session, LoRA use, and reused token count.
5. Contributing to a large OSS project isn't only about PRs. I didn't open one, but a production symptom and a minimal reproduction test moved an existing PR's verification forward.
What I still don't know
- PR #30275 hasn't been merged. Review is still ahead, and its shape may change
- I haven't run this PR in production. Production uses our own fork, a separate implementation of the same approach. The PR was only verified with tests
- I couldn't identify why continuing conversations without LoRA missed 27% of the time before the fix, because the old logs have no decision lines
- The separate effects of the health check fix and the LoRA fix aren't known
These numbers are from GLM-5.3-Flash 3-bit on two RTX PRO 6000 Blackwell cards with four slots and --kv-unified. In particular, GLM-5.3-Flash is hard to roll back when a prompt forks in the middle of the cache, so the cost of a miss varies by model.
If you might have the same symptom
If you specify lora per request and use the RAM cache from --cache-ram, check for these patterns:
found better promptfinds an entry with a high match, but the next prefill covers all the tokens- Only requests with a LoRA miss after their slot has changed hands
Running the test above as is will show whether your version is affected.