We Blamed Our Rules. The 4B Model Was the Ceiling.

Our lead triage sorts estate-agency enquiries on a Mac mini. A decision model, Kev-4B, answers closed questions about each message: what the sender wants, how soon, whether they are qualified. It returns a probability for each option and writes no text. The agency’s written rules, in code, turn those answers into an urgency and a next step. Some rules use cuts, probability thresholds such as “business pitch ≥ 0.60 → spam”. We measured it on synthetic enquiries and published every message.

Every time it got something wrong, we fixed the questions or the rules. A holiday-home buyer read as a holiday renter got a new yes/no question. A business pitch that slipped through got a stricter rule. That went on for three experiments. This week we tested the assumption underneath it: that the model was good enough, and the mistakes were ours.

Same questions, same rules, a bigger model

In exp_047 we put Clef 27B in Kev-4B’s place. It is a decision model of the same kind, about seven times larger (Apache-2.0, run as Q8_0 on llama.cpp on the same mini; exp_041 records the files and licences). Nothing else changed. It got the same twelve questions on the same 566 synthetic enquiries in seven languages, and the same code, with the same cuts, turned its answers into decisions.

Denominators: request 566, next step and qualified 565, timeline 532, genuine enquiries 485, spam 81.

The confusions we had been engineering around mostly vanished:

• valuation read as sell, 10 → 1;
• buyers read as holiday renters, 9 → 1;
• questions read as transactions, 11 → 0.

Polish, our weakest language in every record since exp_043, was Kev’s weakness, not the language’s.

One detail says it plainly. With Kev, our code rules lift its request accuracy from its own 0.883 to 0.919: the rules were doing real work, patching the model. With Clef, the model’s own reading is already 0.991, and the rules change nothing. We had been writing rules to compensate for a reader.

One field went the other way. Financing, where stated, fell from 1.00 to 0.923. All ten misses were enquiries whose mortgage was already approved. Asked “will the sender need a mortgage?”, Clef answered no; Kev didn’t. The wording was wrong, not the enquiry.

Not 27B: two small models, splitting the work

Clef 27B takes 13.5 seconds per enquiry and needs about 29 GB loaded. (The 13.5 s was measured while another model shared the GPU; it was 13.2 s without.) A queue can absorb that, but the mini also runs other models. In exp_048 we tried its smaller sibling, Clef-Flash 9B, on the same 566 enquiries.

Alone, Flash read requests better than Kev did (0.965 against 0.919) and made “qualified” (0.901) and timeline (0.773) good enough to show an agent. But it flagged 9 genuine enquiries as spam: four lawyers writing for a client, two relocation firms, a developer buying land and two other buyers. Two of the nine came from the pitch cut, which was tuned for Kev, not from Flash’s own reading. Flagged enquiries go to a review folder, not the bin, but our spam-safety bar existed to fail exactly this.

Before the test, we had registered a split from what dev showed: Kev answers the request questions, Flash answers everything else, and the same rules combine them. On test, the split kept Kev’s request accuracy and spam safety, and took Flash’s facts.

The split gives up Flash’s request gain (0.046) to stay spam-safe. Its 5.5 s is two calls that should run at once, but both servers share one GPU, so they effectively run one after the other. That split, with the question changes below, is what the CasaSol bot runs now.

Then the errors really were in the questions

With a better reader in place, exp_049 went back to the questions, but only where the recorded errors pointed. The split’s mistakes on earlier test sets fell into three patterns we could target:

• timing given only as a month, date, season or holiday (“from 1 November”, “for Easter”), the most common;
• owners who only want a price, read as sellers;
• clients writing through someone else, read as spam.

The fourth pattern, pitches read as clients, had no closed question to fix it.

Eight writers (Fable 5.1) wrote 560 fresh enquiries, with hard groups built from the three patterns. A blind checker of a different model (Opus 5.5) kept 513, and gemma4:26b, a third model family, labelled them all independently. We tested one change per pattern, plus the financing wording from exp_047. Three of the four had their own pre-registered bar, and the test set had 350 enquiries.

The model didn’t know what day it was. “We’re arriving in November” means within weeks or within the year, depending on today’s date, and nothing told the model the date. We added one sentence to the timeline question: “Today is 21 October 2026”. On the 54 test enquiries written to contain this error, timeline went from 0.444 to 0.870: 23 better, none worse. On ordinary enquiries it rose from 0.737 to 0.817.

One narrow question fixed the false spam. “Is the sender writing on behalf of someone else who is the actual client?”, answered by Flash, now overturns a spam reading. Genuine clients flagged as spam went from 2 to 0 on the fresh set, with spam caught unchanged. On an older sealed test it has a cost: an advance-fee scam that claims to act for “a late client” now reaches the agent instead of the review folder. We kept the rule; missing a real client costs more.

One fix stays off. A “price only?” question corrected six enquiries (four in the valuation group, two ordinary ones) and broke none. It still fell short of its registered bar: +0.074 on the 54-enquiry valuation group, against +0.10 and p < 0.05 (p was 0.06). It stays off until a test with enough valuation enquiries to tell.

The financing question now asks whether the purchase will “use a mortgage or loan, already approved or not”. Financing where stated went from 0.985 to 1.00 on the fresh set.

Two results that stayed small

• Translating first changed nothing we could measure (exp_046). Kev’s model card says English only, so a local model translated 425 non-English enquiries into English before Kev read them. It scored 0.927 on the original and 0.925 translated, with per-language moves of about ±0.03 in both directions. Polish stayed weakest either way.
• Voice notes keep the triage, budgets aside (exp_045). We had computer voices read enquiries aloud, squeezed the audio to WhatsApp quality and transcribed it with Whisper on the mini. The request held (0.912 → 0.900), and so did the next step (0.931 → 0.921). Budgets suffered: two thirds of the losses came from the voices misreading abbreviations and number formats (“M€”, “250k”, “250.000 €”). It also exposed a gap. Whisper writes “siebenhunderttausend” as “700.000”, with no currency, and our budget reader needed a currency or a multiplier word. The next version of the reader handles amounts after a cue such as “up to” or “budget” (its code). Bare amounts with no cue still miss.

What this does not show

• These are synthetic enquiries, written by one family of models and kept only where two readers agreed. They are cleaner than an inbox (we wrote about that yesterday).
• The 566 enquiries behind the 0.991 were labelled by two Fable readers who agreed. A larger model may partly be agreeing with Fable’s reading. Only the exp_049 set has an independent third reader. It agreed with 96 % of the request labels there, and the gains held on that subset.
• The cuts were tuned for Kev and never re-tuned for Clef or Flash. Both bigger models are measured at a disadvantage, and two of Flash’s nine false spams come from that cut alone.
• exp_049’s hard groups were written to contain the errors. They show the fixes work on those patterns, not how often the patterns occur.
• Each version ran once on each test set, and real enquiries remain untested.

What we take from it

1. Before fixing the rules, swap the reader. One run with a bigger model behind the same rules sorted our errors into the model’s and ours. We had spent three experiments on the first kind.
2. Ask each model what it answers well. The 4B reads the request and stays spam-safe; the 9B reads the facts. Together they kept half to seven tenths of the 27B’s gain on “qualified” and timeline, at under half its time and memory. The request stayed at the 4B’s level.
3. Tell the model the date. Any question about timing is a question about today. On enquiries that gave only a month or a holiday, one sentence in the question moved timeline by +0.43. On ordinary enquiries it moved +0.08, about the size of the model swaps for that field (+0.12 for Clef 27B, +0.06 for Flash, on a different set).

So what

What changed because of these experiments:

• CasaSol’s triage now runs two decision models: Kev-4B for the request, Clef-Flash 9B for the facts. Rev 4.2 was deployed in the morning, and rev 5.1 has been live since the late afternoon. The bot now shows “qualified” and the timeline to agents, which it didn’t before.
• The timeline question carries today’s date, and a client writing through someone else is no longer sent to spam.
• No translation layer, and no 27B model loaded for triage.

What it opened:

• Flash alone, now that the on-behalf question exists. It read the request at 0.965 but flagged 9 genuine clients as spam. If the new question removes those, one 9B model may beat the split. Not measured.
• The price-only question, re-tested with about 150 valuation enquiries.
• A guard on the on-behalf rule, so that a scam claiming a client stays in review.
• A call-request rule: “call me, here’s my number” should mean call now. Andrei raised it while reading the test set, and the rules don’t say so yet.
• Real enquiries, the test this whole series can’t replace.

Records, with every message, label, prediction and the code that ran: exp_045, exp_046, exp_047, exp_048, exp_049.