A Local Decision Model Made Our 4B Refuse 92 % of Unanswerable Questions and None of the Answerable Ones

In exp_040 we tried to train our 4B property model to say “not in the material” when the retrieved study text does not contain the rule. Training did not fix it. When the source was missing, the untrained model still answered 55 % of the time, from whatever it remembered.

So we stopped asking the model to know what it does not know. Before the model answers, a separate small decision model reads the retrieved text and the question and returns one probability: can this material answer this question? If not, the system says “not in the material” and the model never answers.

What happened:

• The first test failed. The best gate (Clef 27B) raised correct refusals from 45 % to 95 %, but it also refused 11 % of answerable questions. Our limit was 5 %.
• Every wrong refusal was a retrieval failure. In all 11 cases, retrieval had not returned the source text, so the text the model saw could not answer the question. Most were Spanish questions searched against our English guide.
• We fixed retrieval. We search with an English translation of each Spanish question and take two passages from the best topic. The source text is now found for 87 % of English, 77 % of Spanish and 90 % of Polish questions, up from 78 %, 49 % and 89 %.
• The second test passed. On the same 150 sealed questions, Clef 27B raised correct refusals from 47 % to 92 % and refused none of the 101 questions whose source text was there. Correct answers rose from 75 % to 83 %.
• Two caveats. We reused the test set, so this is weaker evidence than a fresh one. The gate’s probabilities are not calibrated: the threshold works, but the number is not a confidence.

The result in Simplified Technical English

We have a small model. It answers questions about Spanish property law from short passages of a guide. A program finds these passages. Sometimes the passages do not contain the answer. Then the model must say “not in the material”, but it often answers from memory.

We added a second small model before it. This second model reads the passages and the question. It gives one number: the probability that the passages can answer the question. If the number is low, the system says “not in the material”.

In the first test, the second model stopped too many questions that had answers in the guide. We examined them. In each case, the program had not found the correct passage. Thus the second model was correct about the passages.

We improved the program that finds the passages. Then we tested again on the same 150 questions. The second model stopped 92 % of the questions that it must stop. It stopped none of the 101 questions that had their passage. The number of correct results went up from 75 % to 83 %.

exp_041: four local gates

Decision models such as Jev are trained to choose between fixed options and return a probability. They do not write text. Open versions run on llama.cpp’s /v1/systemone endpoint. We tested four gates, all Apache-2.0 and all local on a Mac Mini:

We set each gate’s threshold on 615 development questions so that it blocked at most 3 % of answerable ones. We chose the primary gate on that data, then ran each gate once on our sealed set of 150 fresh questions:

• 38 should be refused: their source text is withheld, or they are off-topic;
• 112 are answerable from the guide.

Everything was pre-registered and signed before the first run.

Clef 27B did what we wanted on the 38 questions it should refuse, and correctness rose 11 points. But it wrongly refused 12 of the 112 answerable questions, above our pre-registered limit of 5 %. Verdict: no gate earns its place.

Two things surprised us:

• Our own model, asked directly, is a poor judge. On eight test rows Qwen3-4B looked perfect. On the development set it caught only 35 % of questions whose rule had been withheld.
• Kev-4B’s card says English only, but it worked about as well on Polish and Spanish.

Why the gate “over-refused”: it was right about the text

We looked at the 11 answerable questions that Clef 27B newly refused. In all 11, retrieval had not returned the source text. The material the model saw did not contain the answer. The untrained model got 9 of them wrong anyway; the gate cost 2 right answers, both answered from memory.

Most were Spanish. Our guide exists in English and Polish, so Spanish questions are searched against the English text. Word-based search then missed the source text for half of all Spanish questions.

So the gate was not the problem. Retrieval was. Our “answerable” label counted what is in the guide, not what the model was shown.

The retrieval fix

We fixed retrieval before testing again. We measured on our 734 training questions, with 29 held-out questions as a check, and did not touch the test set:

The misses usually found the right topic but the wrong passage, because retrieval took only one passage per topic. Taking two from the best topic fixed most of them. Only the final variant improved every language with no regression on the held-out questions. It costs about 110 more tokens per prompt.

We also fixed a small bug that had cut the last line of complete answers.

exp_042: the gate with fixed retrieval

We changed one definition before testing again. A question is answerable only if its source text is in the passages the model saw. The glue’s own rule is “never fill the gap from memory”, so refusing a question whose source was lost is now counted as correct. We count those questions separately.

Everything was pre-registered and signed before the first run. The gates were fixed in advance, and each threshold was set on new development data, built with the fixed retrieval.

Notes on the result:

• The 11 source-lost questions. Retrieval still lost the source text for 11 of the 112 questions the guide can answer (10 Spanish, 1 Polish). Clef 27B refused 8 of them. The untrained model had 5 of those 8 right, from memory. We count those 5 refusals as wrong answers in the overall score, and the score still rose 8 points.
• The retrieval fix helps on its own. Without any gate, the untrained model went from 70.7 % to 75.3 % correct on the same questions. This comparison is descriptive, not registered.
• Calibration failed. We required an expected calibration error of at most 0.10. Clef 27B’s was 0.17 and Kev-4B’s 0.19. The threshold separates well; the raw probability should not be shown to a user as a confidence.

What we take from it

1. Ask a separate model whether the material is enough. The answering model does not do this reliably, even when we trained it to (exp_040), and it is a poor judge when asked directly (35 % on hard cases). A decision model built for closed choices does it well.
2. Check what your “false” errors are. The first gate looked like it over-refused. Every case was retrieval losing the source. A label based on the guide, not on what the model saw, blamed the wrong part.
3. Fix retrieval before you test the model. Half of our Spanish questions never reached their source text. No model can fix that downstream.
4. Small can be enough. Kev-4B, at a sixth of Clef 27B’s time per question, got nearly the same result: one wrong refusal, and +6.7 points correct.

So what

What changed because of these two experiments:

• Retrieval is fixed in the glue. The source text is now found for 87 % of English, 77 % of Spanish and 90 % of Polish questions (it was 78 %, 49 % and 89 %).
• The glue now runs a gate. We chose Kev-4B over Clef 27B: about 2 s instead of 12, and one wrong refusal in 101 instead of none. It runs as a local service on the Mac Mini.
• Each language now has its own threshold, set by the same rule, so Polish users are not refused twice as often as English ones.
• CasaSol Guide, our live Telegram bot, runs the gate in shadow mode. It scores every legal, tax and process answer and logs the result, but blocks nothing yet. The bot finds its passages in a different way from our test pipeline: three long chunks of the original course text. The thresholds do not carry over. We replayed 60 questions that the guide can answer through the bot. At our thresholds the gate would refuse 29 of them. The bot’s own model (gemma4:26b) is also stronger than our 4B, and it already says “I have no information” when the text lacks the answer. So the gate stays in shadow mode. Whether it adds anything in front of a stronger model is our next experiment.

What it opened:

• The answering model still confuses neighbouring rules when the right rule is in front of it. Once it gave the stamp-duty rate (1.2 %) for the transfer tax (7 %); once it gave the deadline for one tax as the deadline for another. The gate checks that the material can answer, not that the answer is right.
• About 23 % of Spanish questions still miss their source text. The gate then refuses them correctly, which is honest but unhelpful.
• A near-miss rule can fool the gate. A deadline for a different tax looks like an answer.

Those are the next experiments.

Records: exp_041 · exp_042.