Here is how we built the test for our lead-triage model. One AI wrote 440 invented estate-agency enquiries and labelled each one: buying, renting, selling, spam and so on. A second AI read every enquiry without seeing the first label, and labelled it again. Where the two disagreed, or the second wasn’t sure, we deleted the enquiry. That was 85 of the 440, nearly one in five. Then we tested our model on what was left and reported 89 %.

None of it was hidden. The rule was registered before the run, and the result said the test was easier than a real inbox. But the enquiries we deleted are the ambiguous ones, the kind an agent has to puzzle over, and a real inbox contains them.

That is one of seven things synthetic data did to our experiments since September. Most of them have run on data written by Fable 5.1, a much stronger model than the ones we test: the answers we fine-tuned a small model on, the items we trained against, the questions we tested with, and the enquiries. We had little choice. There is no public set of Spanish property-law questions in three languages, and we have no real agency enquiries yet.

Laid side by side, the records say something the single posts don’t. Most of what went wrong points the same way: synthetic data is cleaner, easier and more like its writer than the world it stands in for. Here are the seven, each with the evidence and the record it comes from.

1. Trained on a stronger model’s answers, the small model copied the manner and lost substance

In exp_035 we fine-tuned Qwen3-4B on 363 answers Fable had written, to teach it how a Spanish property professional answers: in the asker’s language, citing the source, handing off decisions. It learned the manner: six of seven format targets met. Where the facts it needed were in its prompt, correctness fell from 1.50 to 1.18 out of 2. Answers that open correctly and then continue into invented detail rose from 10 to 28 of 60. A run with 25 times fewer trained parameters did the same damage.

In exp_039 we trained the same base on its own sampled answers, scored by code, with no text from Fable at all. It gained nothing. Its change in correctness was −0.075, with an interval of −0.20 to +0.05, against about −0.3 for fine-tuning on Fable’s answers. That fits the idea that the small model was imitating a stronger model’s confidence. It doesn’t prove it: the interval doesn’t rule out a loss of 0.2.

2. The untrained model already solved most of the training items

For exp_040 we built 2,104 training items with checkable answers from the same study guide. Before training, we sampled the untrained model six times on each. It got 1,281 right all six times and 270 wrong all six times. Only 553 items had any spread, and the training method we used learns only from the difference between good and bad samples of the same item.

After training, exact match rose by 0.067 at the 4-bit quantisation the phone uses (p 0.015), and by 0.013 at 8-bit (p 0.38). Moving the same weights between 4-bit and 8-bit changed about as many answers as training did.

Our reading: a writer working from a guide writes questions the guide answers plainly, and a model with that part of the guide in its prompt usually gets them right. Counting the solved items first is what told us how little room was left.

3. Asked for hard cases, the writers produced cases the old version mostly passed

In exp_043, our lead-triage model’s commonest error was a buyer of a holiday home read as a holiday renter: 10 of its 25 errors on the type of request, triggered by wording like “over for the week” and “flying in”. For exp_044 we asked each of eight writers for six messages built on exactly that confusion: three buyers using holiday wording, three renters using buying wording.

Of the 31 that reached the test set, the old version read 27 right. That is 87 %, against 90 % on ordinary messages. Business look-alikes, the other “hard” group (genuine clients who sound like companies, and real pitches), came in at 83 %. The new version’s fix moved the holiday group from 27 to 29, too small a change to pass the bar we had registered for it.

The one group that was genuinely hard was budgets written in words (“medio millón”, “pół miliona”): the old version read 0 of 29. That difficulty came from a gap we already knew about, since the code had no reader for numbers in words. The writers didn’t find it. A writer can’t see which cases are hard for a particular model. Only that model’s own errors show it.

4. Two readers agreeing gives clean labels by dropping the arguable cases

This is the opening story in full. The same rule dropped 85 of 440 enquiries in exp_043 and 98 of 584 in exp_044. On the enquiries kept, the two readers agreed on the request every time. Andrei read 30 test messages before the set was sealed, and all six of his notes were on fields the rule had already excluded.

So the labels are clean because the arguable 17–19 % was thrown out. Real enquiries will contain those messages, and we don’t know how many. Both readers were also the same model. Where we used two model families as readers, they disagreed more: in exp_039’s audit of 40 training answers for gaming, Fable flagged 5 and Opus 2.

5. The same writer and the same source produce the same test again

In exp_035, once the first test set was used, Fable wrote 30 fresh questions from the guide, without seeing the old set. A wording-similarity check flagged one. Reading the expected answers found that 23 of the 28 in-scope questions tested a rule the old set had already tested, in new words. When exp_040’s test set was drafted, the rule-level check still removed 28 of 285 drafts for overlapping the training items.

The triage series shows the same thing from the other side. On fresh messages, written without access to the earlier set, the old version read the type of request right 0.896 of the time on ordinary messages, against 0.894 in exp_043. That is a stable measurement of how the model does on what Fable writes. It says nothing about how the model does on what agencies receive.

6. “Clearly fake” contact details weren’t

The triage writers were told that about a quarter of messages should include a contact detail, made clearly fake, “for example +34 600 000 1xx or name@example.com”. In exp_043 the writers kept to padded or reserved numbers. In exp_044, with the same sentence, four of the eight writers invented numbers that look real: 26 in all (Polish 6, Spanish 10, Swedish 3, French 7). Our first scan looked only for numbers in international format and caught 11. We published, a stricter scan found 15 more in national formats, and we rewrote the public history so that no commit holds them. The scan now runs as soon as the data is written.

Then we found our own version of the same mistake. A padded number only looks fake. The UK regulator Ofcom reserves 07700 900000–900999 for drama and never allocates it, and Germany’s regulator keeps similar ranges. We found no such range for Spain. The example in our own brief, and the numbers we used to replace the realistic ones, have the format of ordinary Spanish mobile numbers, and some may be assigned to real people: 42 distinct ones in exp_043’s public data and 71 in exp_044’s, plus nine padded Belgian, French and Colombian ones.

All of them are now in Ofcom’s range (earlier commits still hold them). So are fourteen more that the writers had spelled out in words, like “plus three four six hundred zero zero zero one one five”. No digit scan sees those. We found them only when a voice test turned one back into digits. The masker that hides phone numbers from the model has the same blind spot, so the model read those fourteen numbers. Our contact check now treats only Ofcom’s range as fake, so a future writer brief can’t slip back to padded numbers. Re-scoring the published files reproduces both results exactly. E-mail never had this problem: example.com is reserved for examples.

7. What synthetic data did well: comparing versions and finding bugs

  • Comparing two versions. In exp_044 the old and new triage code read the same 331 messages, and we counted the messages where only one was right. Genuine enquiries flagged as spam fell from 5 to 2 of 282, and budgets in words went from 0 to 29 of 29. Those numbers say the new version handles these cases better. They don’t say how often the cases occur in an inbox.
  • Finding bugs. The number reader took “300 m de la plage”, 300 metres from the beach, as a €300 million budget. It showed up in the writer data before any model ran.
  • A regression gate. Every later change to the triage code re-runs the sealed test set and fails if one more genuine enquiry is flagged as spam.

So what

What we changed:

  • We no longer fine-tune on a stronger model’s text. When we train, the model learns from its own samples, scored by code (since exp_039).
  • Before training, we sample the base on every item and report how many it already solves (since exp_040).
  • Test sets are deduplicated by rule, not wording (since exp_035).
  • Both triage records state how many messages the agreement rule dropped, and that the test is easier than an inbox for that reason.
  • Synthetic data is scanned for real-looking contact details when it is written and again before it is published (since exp_044).
  • Synthetic sets compare versions of our code. Claims about real enquiries wait for real enquiries, and that test hasn’t run yet.

What it opens:

  • Hard cases taken from the model’s own errors, rather than requested from a writer, for the next triage test.
  • A second reader from a different model family wherever two readers must agree.

Records: exp_035, exp_039, exp_040, exp_043, exp_044. The triage records include every synthetic message, label and prediction.