For months, most of what this site has built has had the same shape: a team of small models. Each local model answers one narrow question, and code holds everything between them: the joins, the rules and the facts. We wrote it up as a principle. In June we put it in a line we have repeated since: we didn’t upgrade the model, we upgraded the question.
We had tested that idea against one model only once, with the model’s thinking turned off. In exp_050 we tested it on a whole task against one model allowed to think. Part of our case for a team of small models did not survive.
The test
An estate agency’s front desk gets an enquiry. It has to decide what the sender wants and what to do next, pick which of the agency’s homes to offer, and reply in the sender’s language. We built it two ways, with the same rules and the same 60-home inventory, both invented like the enquiries:
- The team of small models.
- Two decision models, Kev-4B and Clef-Flash 9B, answer closed questions about the enquiry. That is our live lead triage.
- Code applies the agency’s rules.
- Flash says what the buyer needs, and code matches it against the inventory.
- A writer model adds one to three sentences, with no numbers allowed.
- Code prints the listings and the closing line from the record, and checks the writer’s sentences.
- One model. gemma4:26b gets the enquiry, the rules written out and the whole inventory, and returns everything in one call.
The team of small models
One model
The team’s writer is the same gemma4:26b, so what differs is the design. In the team, the two small decision models and code make every decision. We registered the bars before the test. Then we ran each build three times on 247 synthetic enquiries in seven languages. The inventory was shuffled in each run, and the third run added two commercial units that fit no enquiry.
A reply counts as wrong if any of these holds:
- it takes the wrong next step;
- it offers a home that doesn’t fit;
- it states a number that is in neither the enquiry nor an offered listing;
- it is in the wrong language.
“Wrong replies sent” counts only the replies a build marked ready to go out, not the ones it handed to a person.
Thinking changed the dev result
On 74 dev enquiries without thinking, the single model sent 24 wrong replies and the team 3. Our rule, fixed before the run, was to adopt thinking if it sent at least 2 fewer wrong replies. On 30 dev enquiries with thinking, it sent 1 instead of 10, and beat the team there too. So the test ran with thinking on, the stronger of the two settings we tried.
Thinking needs room. Our server caps the context at 8,192 tokens. The prompt was 3.8K tokens, the thinking filled the rest, and 4 of the 30 calls stopped before writing an answer. At 16K they finished.
Update, 2 November 2026. On a different task, more room did not hold up as a fix. In exp_060, 22 of 220 verdicts were empty at 8K. At 32K, 11 of them finished and 11 other messages that had answered at 8K ran to the wall instead; the empty answers were repetition loops, and the context change altered the generated text on 115 of 188 answers. The fix that worked was a second call with thinking off when the first returns nothing. The four calls above did finish at 16K; what no longer stands is the general lesson that a larger context is the fix.
This cuts against one of this blog’s older lessons. In April, in Should We Stop Asking Local LLMs to Think?, we argued against thinking:
- What that post saw: an open-ended request (“analyse all tasks and create lessons”) sent the mini into a 24-minute runaway, and turning thinking off fixed it in one second.
- What it concluded: that thinking “is just more cycles for the model to wander”.
- What still stands: its other conclusion, that a precise question in a small context beats a vague one in a huge context.
On a bounded task with a 3.8K prompt, a minute of thinking is what made the single model good.
The result
| Team of small models | One model, thinking | |
|---|---|---|
| wrong replies sent (of 247) | 4.0 % | 6.5 % |
| replies that should go out, sent and right | 76 % | 94 % |
| same decision in all three runs | 100 % | 77 % |
| time per enquiry (median) | 11 s | 70 s |
| with a 4B model in place of the 26B: wrong replies sent | 3.6 % | 22.7 % |
| phone numbers or e-mails that reached a model | 0 in 3,052 calls | sees the raw enquiry |
Registered scoring. A scorer correction, explained below, gives 3.6 % against 6.1 % (p 0.15) and 76 % against 95 %. It changes no verdict.
Safety. Our bar was that the team sends at most half as many wrong replies, with p < 0.05. It sent fewer, but the difference was not significant (p 0.16). So the team is not shown to be safer than one model that thinks, and the single model is not shown to be safer than the team.
Usefulness. Our bar was that the team gets within 10 points of the single model. It was about 18 points behind.
- Most of the gap is one threshold. The team hands a lead to a person when Kev-4B is less than 0.5 sure what the sender wants. That held back 28 of the 157 leads that should have gone out.
- Below 0.5, Kev was usually right: 83 % of the time. Its confidence runs low.
- We fixed that cut before the test. On dev, a cut of 0.35 would have sent far more right for one more wrong reply, but our rule kept 0.5.
What only the team did
The same decision, every time. The single model changed its decision on 56 of 247 enquiries between runs:
- 22 changed only which homes it offered;
- 18 changed only the next step;
- 14 changed whether it sent a reply at all, most of them along with something else;
- 2 changed both the homes and the next step.
The team changed none. Its writer worded its sentences differently each run, but the facts in the replies were the same.
Errors with an address. With the corrected scoring, the team’s wrong replies were the same 9 enquiries in every run, so each one points at a part to fix. The single model’s fell on 22 different enquiries across the three runs, and only 5 were wrong all three times.
Team of small models
0 of 247 decisions changed between runs. Wrong replies on 9 leads, the same 9 in every run.
One model, thinking
56 of 247 decisions changed between runs. Wrong replies on 22 different leads, 5 of them in every run.
Facts come from the record. The single model wrote its own listing lines and misquoted two of them: 135 m² for a flat of 155, and 860,000 € for a townhouse listed at 1,090,000. The team prints its listing lines from the inventory.
A code check on the single model is not enough. We also passed the single model’s replies through the team’s final check, offline. It caught wrong languages and invented numbers, and it left 14 wrong replies against the team’s 9 (p 0.20). A wrong next step or a home that doesn’t fit looks fine to a check on the text, because those are decisions, not wording.
The writer is a replaceable part. With a 4B writer, the team did as well, within one lead. The single model on the same 4B sent 23 % wrong replies.
Contacts stay out of the models. Phone numbers and e-mails are masked before any model sees the text. None of the test’s contact details turned up in any of the 3,052 model calls the team made across its four runs. Names do get through: they are part of the text the models read.
Speed. 11 seconds against 70. In a queue that drafts replies for an agent, 70 seconds may be acceptable. For a reply while the client waits, it is slow.
What we got wrong in our own team
A blind judge (Opus, shown both builds’ replies, shuffled) said a careful agent would send 31 % of the team’s replies unchanged, against 43 % of the single model’s. The team’s two most common faults were bugs in our own fixed text:
- The callback promised twice (27 replies). We told the writer to say an agent would call, and the closing line said it again.
- “Would you like to see any of them?” after a single home (20 replies).
Our final check looks at language, numbers and contact details, so it saw neither. One code change fixes both. The single model’s most common faults come from the model, so they can’t be fixed in one place:
- English words left in non-English replies (34, in six of the seven languages, most often Polish and Swedish);
- raw listing data pasted as text (12).
Two more of our mistakes were caught on the way:
- Our first test pass recorded server failures as the writer’s own. A second model we had left loaded squeezed memory, and Ollama evicted the writer. Its incomplete responses went into the record as writer failures. We stopped after 12 enquiries, archived the pass unscored and restarted. The harness now retries an incomplete response as a server failure.
- Our scorer had a false positive against each build. It read the French “grand plaisir” as 1,000 (English slang, “a grand”). And when a sender spelled out a phone number in words and the reply wrote it in digits, it counted the number as invented. Correcting both changed no verdict, and both scorings are published.
One kind of “wrong” was a good reply. Six of the single model’s wrong replies declined a holiday-let request in the sender’s language: “we do not offer holiday rentals”. Our rule says such leads go to a person. Counted as right, they would leave both builds level on wrong replies, 9 against 9 (corrected scoring).
Looking back over fifty experiments
exp_050 was meant to test the whole approach at once. Here is where our earlier claims stand:
- “We upgraded the question”: a correction.
- In a post we rewrote in June, we said that a capable model, handed a flat question about seven listings, produced five false positives. The record (exp_016C) shows that the flat arm was a database filter with no model in it.
- Our one real comparison of this kind was exp_012 against exp_013. On a document-audit task, gemma4:26b scored 0/8 alone and 2/3 inside the structure. The structure won, with the model’s thinking turned off.
- exp_050 is the first such comparison with thinking on, and the single model held its own. We have added dated notes to that post, to the cost-capability post, and to the April post on thinking.
- Small models as parts: holds. On this set, Kev-4B read the request right 0.947 of the time. Flash answered 2,240 questions about areas and features with 0.957 accuracy, and its confidence matched how often it was right. The writer could be swapped for a 4B one.
- Training a small model into the behaviour: not tested here. The three training runs stay a negative result, and code around an untrained model stays our build.
- Synthetic data: still the limit. Everything here is invented, and it is cleaner than an inbox.
The case for a team of small models now rests on reliability, not capability:
- every decision is repeatable;
- every error points at a part to fix;
- listing facts come from the record, and a code check on one model’s text cannot catch a wrong decision;
- the parts are small, fast, and keep phone numbers and e-mails out of every model.
On this build, the usefulness gap was about 18 points, and most of it is one hand-off threshold we fixed before the test.
What this does not show
- Synthetic enquiries and an invented inventory, written by one family of models and kept where a second agreed.
- One design per build. The team’s fixed text was not revised after dev, and the judge found two bugs in it. A revised team would score higher; this test doesn’t show by how much.
- The agency’s rules are part of the score, as the holiday lets show.
- The judge was not truly blind: the team’s fixed format is recognisable.
- One single model on one Mac mini: a mixture-of-experts with about 4B parameters active. A larger single model, or a cloud one, was not tested.
So what
- Nothing changes in production. The triage stays as it is, and no reply drafter goes into the bot from this.
- The next test is the two builds checking each other. In the run we compared, they never sent a wrong reply on the same enquiry. Sending the team’s reply only when the single model agrees would have cut wrong replies from 9 to 2 (corrected scoring), with almost no loss in usefulness. We found that after the test, so it is registered as the next experiment.
- Also open:
- Kev’s hand-off cut, set on fresh data;
- the two template bugs, fixed and judged again;
- the cost-capability test re-run with thinking on;
- real enquiries, which no synthetic set replaces.
Every enquiry, label, prompt, prediction, both scorings, the judge’s verdicts and the aborted first pass: exp_050. The figures are built from those files by figures50.py.