Our 4B Property Model Gained 6.7 Points at Q4 and None at Q8

In exp_039 we trained Qwen3-4B with GRPO against a reward that checked form: language, citations, numbers, length. The model learned what the reward paid for and did not get more correct. The post ended with a list of changes. exp_040 makes the main one: the reward checks the answer itself.

Every answer now ends with one line, Answer: …, and code checks that line against a closed label: one of the question’s options, a number computed in code, or “not in the material”. A right line earns +3, a wrong or hedged line −2, a missing line −1. Language, plain text, citations, number support, length, repetition and copying stay as small terms.

What happened:

• Our audit found the reward not gamed. Two blind readers each read the 40 highest-scoring training answers. Fable found 0 gamed and Opus found 1, under the pre-registered limit of 4. exp_039’s first run failed this audit at 5.
• At Q4 the trained model looked better. On 150 fresh sealed questions, exact match rose from 0.707 to 0.773: +0.067, 95 % CI +0.013 to +0.120, McNemar p 0.015.
• At Q8 the gain was gone. 0.727 → 0.740: +0.013, CI −0.027 to +0.053, p 0.38. The pre-registered verdict is not robust.
• Quantisation alone moves as many answers as training did. The same weights at Q4 and at Q8 disagree on 17–19 of the 150 questions. Training changed 18.
• Behind the glue, trained and untrained score the same: 0.733 each. The deployed build does not change.

The result in Simplified Technical English

We have a small model, Qwen3-4B. It answers questions about Spanish property law.

We trained it with a new score. The model writes an answer and then one last line with the short answer. A program compares the last line with the correct answer. A correct line gets a high score. A wrong line gets a low score.

A strong model wrote the training questions and their correct answers. A second strong model checked each question without seeing the answer. We kept only the questions that the small model sometimes answered correctly and sometimes not. 622 questions stayed.

Two strong models read 40 high-score answers. They did not see the scores. They found 1 bad answer in 40. The limit was 4. Thus the score was safe.

Then we tested on 150 new questions. Nobody in the training work saw these questions. We compressed each model in two ways: strong compression (Q4) and light compression (Q8).

With strong compression, the trained model was better by 6.7 points. With light compression, it was better by only 1.3 points, and this can be chance. When we only change the compression of one model, about 18 answers change. Training also changed about 18 answers.

Result: we cannot say that the training made the model more correct.

The setup

Unchanged from exp_039:

• the base (Qwen3-4B, snapshot 1cfa9a72);
• the adapter (LoRA rank 16 on the four attention matrices, 11,796,480 parameters);
• the trainer (mlx-lm-lora 5.8.6, group 6, KL leash 0.02, clip 0.2, temperature 0.9, 512 tokens);
• retrieval and the prompts.

What changed:

• The quantiser first. exp_039’s apparent gain came from our Q4 recipe damaging the untrained base. Step 0 built an importance matrix from the guide text and re-quantised the base. Its Q4 now scores within 0.025 of Q8 on the old set (1.375 vs 1.40). Every exp_040 arm uses this recipe.
• Labelled training items.
  • Fable wrote 734 questions from our property guide, each with a closed label, in English, Polish and Spanish.
  • A second, blind Fable instance answered each one from the guide text alone. Disagreements were dropped.
  • About a third come as contrast pairs: one fact changes and the answer flips.
  • We added 163 abstention items, whose source is removed from retrieval and whose answer is “not in the material”, each confirmed by a blind reader. We also added arithmetic with gold computed in code, and off-topic requests.
• Offline dynamic sampling. The untrained base answered every prompt 6 times. We kept only prompts it got right 1 to 5 times: a group where all 6 answers score the same gives GRPO no gradient. 622 of 2,267 prompts survived, so training was 78 steps, 4 h 44 m on a MacBook Pro M5 Max.
• A new sealed test set, v3. 150 questions built in a separate session that never saw the training work, from guide sections held out of training.
  • Composition: 30 contrast pairs, 52 single questions, 23 abstention questions and 15 out-of-scope questions; 50 each in English, Polish and Spanish.
  • Andrei read 30 of them before the seal. Scoring is exact match by code, with no judge.

Everything was pre-registered and signed before training started.

What the numbers say

At Q4 the gain looks like what we hoped for: contrast pairs with both twins right rose from 0.50 to 0.70, Polish from 0.66 to 0.82, and abstention up. At Q8 almost all of it disappears.

The question is how much quantisation alone moves. Paired over the 150 questions:

At 4 bits the quantiser decides about one answer in eight on its own. A training effect of a few points sits inside that churn. The pre-registered Q8 arm is what tells the two apart.

Why training did not move more

• The base already knew most of it. With the source text retrieved and the options shown, the untrained model got 71 % of the categorical training questions right 6 times out of 6. The filter dropped those, so training saw 622 rows, not the 1,700 we planned.
• The training reward was flat. Block means of the answer term ran 1.09, 0.94, 1.02, 0.87, 1.47, 0.87. exp_039’s form reward climbed steadily, but form is easier to learn than correctness.
• Checkpoint selection had nothing to select on. On the validation set the base was already 78 of 79.

The model can choose between options when the rule is in front of it. Where it fails is reading a rule into a new case, and abstaining when the rule is missing. 600 questions and 78 steps did not change that robustly.

What we take from it

1. In our audit, a reward that checks the answer was not gamed; the form reward was. This is the part we’ll keep.
   • The reward reads only the final line, so it cannot see whether the explanation above it is right. The one answer Opus flagged argued “no” and then answered “yes”.
   • We did not run the judged check of the explanations.
2. At 150 questions, Q4 is not a measurement instrument for a few-point effect. The next training cell either measures at one quantisation chosen in advance, with a larger test set, or it doesn’t claim a gain.
3. The deployed build stays the same: the untrained base behind the deterministic glue. That glue already decides what training was meant to teach: when to decline, when to hand off, which rule to show.

Full record, code, aggregate results and the audit: exp_040 on GitHub. Weights, released whichever way the result went: miktam/coapi-voice-r3-gguf.