Introduction
I build kotonia, a character chat service, on my own. Its local LLM runs on two GPUs at home. The marginal cost of local inference is mostly electricity; that doesn't erase the cost of hardware, maintenance, or my time.
Revenue for October 2026 was about ¥6,000. One evening, I sat down with an AI to work out how to grow it.
Here was the hypothesis I brought:
From playing competing services, I felt they handled darker romance scenarios well. My model tends to steer toward happy endings. Instead of dedicating the GPUs to a large model, should I use LoRA to tune a smaller one toward the writing style I want?
That hypothesis eventually fell to the bottom of the list without ever being tested. Arguments didn't move it there. Logs did.
How the priorities changed
Over one evening, our top priority kept moving:
| Evidence supplied | What became the top priority |
|---|---|
| My hypothesis and impressions | A smaller model plus LoRA |
| Usage records for 12 people who had paid | Retention after payment, custom characters, and voice |
| Revenue for October 2026 | Acquisition and onboarding rather than the model |
| The funnel for 1,254 signups | The entrance: half had never sent a message |
| Behavioral logs for people with zero turns | The 414 people who saw a greeting and left |
| The screen's code and individual database rows | A one-line display condition |
The signup funnel covers people who registered between August 28 and October 28, 2026, excluding repeat registrations from the same device and excluded or suspended accounts. The 12 lifetime paying users are a different population.
The further down the table we went, the smaller and more concrete the task became. It also became cheaper to fix.
Acting on the first row could have meant weeks of model training. At the bottom, we fixed things like these:
- Opening reply suggestions never appeared for characters with a greeting, although 75 of 102 public characters had suggestions configured.
- A self-introduction modal appeared before the user's first message. Even 82 people who filled it in and saved it left without saying anything.
- Our “first turn” event fired on the character's greeting. It had been recorded for 413 people who had never sent a message.
None was a difficult fix. The difficult part was finding out they existed.
AI can reason only from the evidence it has
The first AI updated its recommendation whenever I supplied another fact. Mention a paying user with 1,000 turns, and it valued long-term immersion. Mention ¥6,000 in revenue, and it emphasized acquisition. Wonder aloud whether the character catalog was too small, and it suggested a discovery feed.
Every answer made sense within its frame. The problem was that the order in which I supplied facts became the order of its recommendations. The AI had to work from the facts I brought to the conversation, and those facts were organized around my own concerns. A developer worried about model quality gets a conversation about model quality.
I brought the same problem to a second AI with access to the code and database. Its first move was to write SQL. It split the signup cohort by source and traced what people with zero turns had actually opened and clicked.
The results weren't in my hypothesis:
- ChatGPT was the largest acquisition source, with 413 signups. Only 102 were from Japan; the users from other countries had a low rate of reaching ten turns.
- Only 19% had visited the catalog. The default character accounted for 75% of first conversations.
- The 414 people who saw only a greeting hadn't all closed the page immediately. Their median time on the conversation screen was 69 seconds.
Sixty-nine seconds without sending anything looked more like “I don't know what to say” than an instant rejection. That's an inference, not something we asked users, and dwell time alone cannot establish the reason. But it gave us a reason to investigate the entrance before changing the model's writing style.
Give intelligence more of the outside world
One idea keeps coming back in my work as an engineer: getting more out of intelligence often starts with supplying more external evidence, rather than trying to make the intelligence itself better.
For people and AI alike, reasoning ability determines what can be inferred from the material available. If the material is missing, a clever reasoner can still produce a plausible mistake very quickly.
That happened three times that evening.
First, it happened to me. I was about to choose a direction based on my impressions of competitors and my frustration with our model.
Second, it happened to the first AI. It kept producing reasonable conclusions from the evidence I had supplied.
Third, it happened to the AI investigating the implementation. While adding a reply-drafting feature, it confidently chose a separate, short prompt because including the full conversation would supposedly disrupt the cache. It hadn't measured that assumption. I asked why we couldn't reuse the conversation's normal prompt path. A ten-minute experiment overturned the design: on synthetic conversations, calls taking 5–12 seconds fell to 1.6–2.6 seconds. I describe that investigation in the cache article.
Immediately afterward, it inferred from one run that the cache would fully recompute after the fourth helper call. I asked whether a checkpoint should have been restored. It read the source and corrected the explanation: the cause was a different rule, not the call count.
Each improvement started with a short question. Those questions didn't require extraordinary intelligence. I had seen the actual behavior and noticed that the explanation didn't fit.
What I now do differently
Pull the available logs before asking for opinions. Bring a hypothesis and the discussion orbits it. Bring numbers and the discussion follows what they reveal.
Read an actual row after hearing the conclusion. A zero-turn session contained exactly one row: the character's greeting. That row exposed both the display bug and the broken event definition. The aggregate alone didn't.
Ask “Did you measure it?” Especially when the explanation is elegant. An elegant explanation is easy to write without measuring anything.
Question the instruments too. Our first-turn event had counted greetings for months. I can no longer reconstruct every decision influenced by that number.
Ask what remains unverified. A useful report states where the evidence stops, as well as which checks passed.
Where we ended up
We didn't change the model or the GPU configuration.
We changed a display condition, the timing of a modal, catalog card headings, and a button for drafting replies. As of November 12, 2026, we haven't measured the effect on user behavior. We'll rerun the same SQL after two weeks. Because several changes shipped on the same day, a before-and-after comparison won't isolate each one's effect. If the numbers don't move, we'll inspect the next set of logs.
Talking to a clever partner is enjoyable. That evening, the most useful thing I could give one was a number nobody had looked at yet.
