A Local Decision Model Made Our 4B Refuse 92 % of Unanswerable Questions and None of the Answerable Ones

In exp_040 we tried to train our 4B property model to say “not in the material” when the retrieved study text does not contain the rule. Training did not fix it. When the source was missing, the untrained model still answered 55 % of the time, from whatever it remembered. So we stopped asking the model to know what it does not know. Before the model answers, a separate small decision model reads the retrieved text and the question and returns one probability: can this material answer this question? If not, the system says “not in the material” and the model never answers. ...

October 18, 2026 · 10 min · Nestor

We Trained It Three Times. Then We Stopped.

Correction, 24 October 2026. The glue met 5 of the 7 format floors on the 60 questions, not 7, and two of its floors failed on the fresh 30-question set this post promised to report. Format floors. The table gave the glue 7 of 7. Its judge file records 5. It missed answers citing (43 of 60, floor 90 %) and artefact rows (15, floor at most 5). Source: designA_A12_judge_20260922.json, field reference_recommendation.preregistered_floors_status. Against v2. v2 met both of those floors: 55 of 60 answers cited, and it had 2 artefact rows (b3_v2_judge_20260922.json). The glue did reach the two floors no fine-tune reached, but it did not take “every format floor the fine-tunes bought”: it lost these two. v2’s count. The record’s summary table gave v2 6 of 7. Its v2 judge file and verdict give 5: v2 missed out-of-scope (2 of 3) and Spanish term (89.6 %). The table and the v2 paragraph now say 5. The fresh set. The one re-score ran on 13 October, on 30 sealed questions the glue had never seen. Two floors failed: out-of-scope declines 0 of 2 and language 29 of 30. Both out-of-scope questions passed the scope gate and got invented answers. Plain text (30/30), hand-off (28/28), numbers (98.9 %) and citations in context (97.2 %) held. Under the pre-registered rule those two floors were fitted to the first 60 questions, and the claims now stand at 0/2 and 29/30. The judged half of the re-score is not in the record yet. v1 citations. 98 % was the share of v1’s citations found in the retrieved text. 50 of 60 v1 answers cited anything. Q8_0. The Q8_0 diagnostic ran on v1, not v2. The checkpoint and temperature diagnostics ran on v2. Record: HYPOTHESIS.md (blocks “Diagnostics after B3 v1”, “A12 judged” and “A12 on eval v2”) ...

September 22, 2026 · 16 min · Nestor