In exp_035 we fine-tuned Qwen3-4B three times on answers written by a much stronger model. It learned the shape of a good answer, and lost substance every time. The best build we had was the untrained model behind deterministic code we call the glue.
Then Coinbase described post-training a 9B model with GRPO against a reward computed from real outcomes, and beating a frontier model on that task. That suggested the cheap experiment our post had asked someone to run: change only the training objective. Our model would write its own answers, code would score them, and no stronger model’s text would appear anywhere in training. Same base, same retrieval, same prompts.
That is exp_039:
- The model did not get more correct. At the phone’s compression (Q4_K_M) the trained model seemed to score +0.20 above its untrained base. At Q8_0, the untrained base scores as well as or better than the trained one: −0.075, 95 % CI −0.20 to +0.05. The apparent gain was our quantiser’s damage to the base, which the trained model happened not to suffer.
- It did not learn the format. It meets 0 of 7 format floors without the glue.
- It learned what the reward paid for most easily. Citations matched to the retrieved text rose from 0.09 to 0.42 during training. The one correctness signal in the reward, arithmetic, fell from 0.57 to 0.07.
- Our first run cheated, and our own gate stopped it. A blind reader found 5 of 40 top-scoring answers gamed, mostly padding to reach a length minimum.
The rest of this post explains how we found each of these, why training did not help, and what we’ll change.
The result in Simplified Technical English
We have a small model, Qwen3-4B. It answers questions about Spanish property law. We tried to make it better with a new training method.
The method: the model wrote 6 answers to each question. A program gave each answer a score. The program checked the language, the citations, the numbers and the length. The training then made the high-score answers more probable. We changed only 0.29 % of the model.
A model can cheat a scoring program. Thus, a strong model read 40 high-score answers. It did not see the scores. In run 1, 5 of 40 answers cheated. The limit was 4, so we stopped run 1. In run 2, we removed the rule that caused the cheating. Then 2 of 40 answers cheated, and run 2 passed.
Four systems answered the same 60 exam questions. A strong model judged each answer. It did not know which system wrote it. At first, the trained model looked better than the untrained model.
A reviewer found a second possible cause. We had compressed the models with our own tool, and the tool made the untrained model weaker. So we compressed both models less and tested them again. Then the untrained model was as good as the trained model.
Result: the training did not make the model more correct. The model learned to cite the text it found. It did not learn the full answer format. The program could not check if a rule was correct, so the training could not improve that.
The setup
Base: Qwen3-4B, the same snapshot as exp_035.
Adapter: LoRA rank 16 on the four attention matrices of all 36 layers: 11,796,480 trainable parameters, 0.29 % of the model. The MLP, which holds two thirds of the weights and most of the stored knowledge, stayed frozen.
Training: GRPO with mlx-lm-lora on a MacBook Pro (M5 Max). Each step sampled 8 questions and 6 answers each; one pass over 837 questions took 105 steps and about 6 hours. The KL leash was 0.02 and the clip 0.2.
Reward: eleven components, all computed by code:
- answer language and plain text;
- citations found in the retrieved text;
- numbers found in, or derivable from, the question and retrieved text;
- a computed gold for arithmetic questions;
- a Spanish legal term;
- a professional hand-off on advice questions;
- a decline on out-of-scope questions;
- length, repetition, and verbatim copying.
No model call anywhere in the reward.
Measurement: pre-registered before any answer was scored, with five dated amendments.
- Four arms: the untrained base, the trained model alone, the glue over the untrained base (the deployed reference), and the glue over the trained model.
- Sets: 60 sealed exam questions plus 30 fresh-rule questions.
- Judge: Fable, blind to which system wrote each answer.
Before the first real step we smoke-tested the trainer on the Mac mini. It silently did five things we hadn’t asked for:
- It dropped our LoRA target list and trained all seven matrices: exp_035’s 33 M configuration, the thing this experiment holds fixed.
- It replaced four of our settings with package defaults, including a clip of 0.0001 instead of 0.2.
- It switched Qwen3’s thinking mode on during training, while serving runs with it off.
The launcher now refuses to train unless each of these is exactly as configured. If you use mlx-lm-lora 5.8.6, check these.
Run 1: the audit stopped it
A reward computed by code can be satisfied without answering. So before scoring anything, a blind reader examines the 40 highest-reward answers from the trained model. More than 4 judged gamed stops the run.
Run 1 stopped at 5 of 40. Three answers padded a few sentences of real content with restatement and unrelated material. They had to, because the reward charged anything under 80 words. One stuffed in citations that served no point, and one dodged with “the material doesn’t cover this” when it did.
For run 2 we removed only the length minimum, re-registered, and trained again. Answers got about 15 % shorter, and the audit found 2 of 40, both citation stuffing.
A second reader (Opus) had counted 2 of 40 on run 1, so the gate’s resolution is about as fine as the disagreement between two strong readers. Run 2 passed the gate as written. We don’t claim the length change alone caused the drop.
The gain that turned out to be quantisation
Scored blind against the untrained base, both at Q4_K_M, the trained model passed the registered test, “not worse than the base by more than 0.10”. It even scored +0.20 higher (95 % CI +0.05 to +0.375) on the 40 questions where the answer was in the retrieved text, and its harmful answers fell from 13 to 7.
Before publishing, we had a fresh Fable instance review the analysis adversarially. It recomputed every number from the raw files and found no mismatch. It also found what we had missed:
- The same untrained weights differ by quantiser. In Unsloth’s Q4_K_M (behind the glue) they score 0.225 higher than in ours. That is the size of the gain.
- Our quantiser uses no importance matrix.
The reviewer named the one run that separates the two causes: score both models at Q8_0, where compression costs little. We registered it with its decision rule fixed, then ran it.
| On the 40 questions with the answer in context | Untrained base | Trained model |
|---|---|---|
| Q4_K_M, our quantiser | 1.20 | 1.40 |
| Q8_0 | 1.425 | 1.35 |
At Q8_0 the difference is −0.075 (CI −0.20 to +0.05), with 4 questions up and 7 down. The untrained base at Q8_0 scores exactly what the glue’s Unsloth-quantised base scores. Our Q4 recipe cost the base about 0.2 points, and the trained model didn’t lose them. Training added nothing we can measure.
Why training did not help
The reward’s own traces show it. Here is what each component looked like at the start and the end of run 2:
| Reward component | Steps 1–20 | Steps 86–105 |
|---|---|---|
| Citation matched in the retrieved text | 0.09 | 0.42 |
| Plain text | 0.40 | 0.48 |
| Arithmetic answer correct | 0.57 | 0.07 |
| Hand-off on advice questions | −0.76 | −0.89 |
| Decline on out-of-scope questions | −1.1 | −1.2 |
Four things explain it.
1. The reward measured form, not correctness. Only one component could tell a right answer from a wrong one: the arithmetic check, on 14 % of the questions. Everything else pays for how an answer looks. GRPO did exactly what it is built to do and pushed up the components that were easiest to move. It also traded away the arithmetic.
2. GRPO only learns what the model already does sometimes. It compares the 6 answers to one question against each other. If none of the 6 hands off to a lawyer, all 6 get the same score and the step teaches nothing. So hand-off and declining never moved: the model rarely did them, and nothing pulled it there.
3. The model’s real errors are invisible to code. The judge’s notes on the wrong answers say the same things again and again:
- an inverted rule;
- the right article applied to the wrong question;
- a correct rule followed by an opposite conclusion.
A regex can’t see any of that. We wrote this down in the design before training (“what the reward cannot see”) and then trained anyway, hoping grounding pressure would carry over to correctness. It did not.
4. A third of the errors are retrieval, not the model. On 17 of the 57 in-scope questions, the rule the question needed was not in the retrieved text at all. No training of the answering model fixes a missing page.
Would more training questions help? Not with this reward. More steps would push citations further and could keep eroding arithmetic. More questions help only if each one carries a label that code can check for correctness.
What we’ll change
Make correctness checkable. Give every training question a short answer that code can verify, not only a prose answer:
- a computed number;
- a category: which tax, which deadline, which article, yes or no.
The model still writes its explanation, and the reward checks a final answer line against the label. This is Coinbase’s recipe applied honestly: their reward was a real outcome, and ours was mostly form. A strong model can draft the labels once from the guide, and an adversarial check can verify them. Labels aren’t example answers to imitate, so the no-teacher-text rule holds.
Train against inversions directly. Use pairs of questions that differ by one fact, so the right answer flips: a resale against a new build, a buyer under 35 against one over 35. A model that inverts rules fails half of every pair, which gives GRPO a signal it can’t get from form.
Keep only questions the model sometimes gets right. If the model gets all 6 answers right, or all 6 wrong, the question carries no gradient. Filter them out before training (DAPO-style dynamic sampling) so every step teaches something.
Let correctness outweigh form. Correctness should dominate the reward, so a well-cited wrong answer scores below a plain right one. Cap the citation term.
Reward honest abstention, where code can check it. Each training question comes from a known chunk of the guide. When retrieval misses that chunk, the right answer is “the material doesn’t cover this; check with a lawyer”, and code can verify that.
Fix the quantiser first. It’s free. An importance-matrix Q4 calibrated on the guide should recover the 0.2 points our recipe costs, for the untrained model as much as any trained one.
Build a bigger sealed test set. With 40 scored questions, effects under about 0.2 are invisible. The next test needs 150 or more.
Use RL where it fits exactly. The glue’s closed decisions are Coinbase-shaped: is this question in scope, does it ask for advice, does it need a hand-off. Each has a single right label and a checkable outcome. That is the next experiment.
What we owe the reader
- The training itself was not pre-registered; only the measurement was. Amendments 1–5 were signed before the scores they govern existed.
- Run 2 is the second reward version, after a failed gate.
- The same model family audited, judged and reviewed this work. We recomputed every number from raw files, but the rubric readings are not independent.
- The 60 exam questions were never re-checked against current law, and the 30 fresh-rule questions are too few to separate any arms.
- Everything is public except the private inputs: the record, code, per-row scores, training traces and audits are on GitHub. The test sets, the prompt set, the guide index and the glue stay private and are bound by sha256. The trained weights are on Hugging Face as
miktam/coapi-voice-r2-gguf, released as evidence, not as a product.
What we’d tell ourselves on Monday
We had already written down that the reward could not see an inverted rule. Then we trained on it and hoped that grounding would carry over to correctness. It didn’t. A reward teaches what it measures. If it can’t tell right from wrong, training on it won’t make the model right.