The usual advice for coding agents is to write the test first and let the agent work until it passes. We ran that set-up thirteen times on real work, with a second agent reviewing each result, and kept a table of the defects. The test written first caught almost none of them, and was often wrong itself. Three things follow for anyone who works this way:

  1. Check the test before the work starts. The agent that wrote the task and its test was itself a large source of errors, and the test was wrong in about half the runs, usually by passing while the thing it guarded was broken.
  2. A test for a fix has to hold both sides. In one run a fix passed its tests by stepping aside from the question: the wrong answers were gone, and so were many of the right ones.
  3. Give the reviewer the result, not the worker’s account of it. A reviewer who was not given the worker’s report found most of what was found, including false sentences in results whose numbers were all correct.

The numbers and their limits follow. The agents are Anthropic’s Claude models, not local ones. The counts, the table behind them and a script that recomputes them are public.

The set-up

Three agents and one person.

  • The task-writer picks a piece of work, writes down what “done” means, and writes a test for it. The test is committed before any work starts and has to fail on the code as it stands.
  • The worker does the work against that test. It cannot edit the test.
  • The reviewer gets the result and the task, never the worker’s report, and answers yes or no with objections. Objections that hold go back to the worker once.
  • A person decides everything that leaves the machine: push, deploy, publish.

Thirteen runs closed between 6 and 10 November: six on code, three on experiment kits, one grading run and three on written results. Seven of them built, graded or wrote up two experiments already on this blog, the prompt-injection test and the values fine-tune. The table holds 51 rows: every blocking finding, every defect a run’s verdict names, and every error in a task or a test. A row can group several findings of one kind from one run, and smaller review points are not listed, so 51 is a count of rows, not of every flaw.

1. Check the test before the work starts

passed while the thing it guarded was broken (5)failed on a correct result (2)verdict not found wrong (6)
run 01 (code): the test's verdict was not found wrong01run 02 (code): the test's verdict was not found wrong02run 03 (code): the test passed while false03run 04 (experiment kit): the test failed on a correct result04run 05 (experiment kit): the test's verdict was not found wrong05run 06 (grading): the test's verdict was not found wrong06run 07 (text): the test's verdict was not found wrong07run 08 (code): the test passed while false08run 09 (code): the test passed while false09run 10 (experiment kit): the test passed while false10run 11 (text): the test's verdict was not found wrong11run 12 (code): the test passed while false12run 13 (text): the test failed on a correct result13
One square per run, in order. In 7 of 13 runs the test that was written and frozen before the work turned out to be wrong: 5 times it passed while the thing it guarded was broken, 2 times it failed on a correct result. A grey square means the test's pass or fail was not found wrong, not that the test was proved right: in runs 01 and 11 the reviewer still found cases that could not fail.

The test gave a wrong verdict in 7 of 13 runs. In two more, runs 1 and 11, the reviewer found test cases that could not fail, though they hid nothing.

  • Five times it passed while the thing it guarded was broken. One test of a duplicate rule fed the same bytes twice, so the rule never ran. One asserted a new label and never searched for the words. One used a stand-in for a model that gave the same answer whatever it was shown, and passed 16 of 16 while the property it was written to prove was false.
  • Twice it failed on a correct result. One case’s own test data contained the words it banned. Another looked for a line by a prefix copied from someone’s shortened quotation of it, so it could never pass.

The source matters more than the count. Of the 51 defect rows, 19 are the task-writer’s own: 11 errors in the task and 8 in the test. In three of the five runs where the test passed wrongly, the test had the same blind spot as the task it was written from. One agent had written both.

Writing lessons down after each run did not stop it. The first lesson about tests came out of run 1, and a wrong test still appeared in run 13.

2. A test for a fix has to hold both sides

A week later a larger version of the loop was pointed at a public chat bot that answers property searches from a set of listings. Twelve reviewing agents read and ran the code, a second agent tried to refute each finding rated high or medium, and 90 findings stood, each with a test that failed on the old code. Here the tests were written by the agents that found the faults. Two workers fixed the code against 63 of them; the other 27 were set aside for a later run or for the owner’s decision.

One part of the bot answers a search in code, from the listings, and steps aside when it cannot read the question, which then goes to the language model without the list of homes.

After the fix, the tests for that part stood at 253 of 254. The reviewer did not rely on that number. It ran the old code and the new side by side on 72 everyday wordings of a search, each with one extra detail.

after the fixafter the fix: 4949of 72after one more roundafter one more round: 55of 72
Everyday searches with one extra detail, out of 72, that no longer got the list of homes the old code gave. The frozen tests for this part read 253 of 254 both times. The second count is the task-writer's re-run of the reviewer's comparison.

On 49 of the 72, the new code no longer gave the list of homes the old code gave. On 107 plain searches with a single filter, 4 lost the list and 3 gained one; the reviewer’s estimate was that a quarter to a third of real searches would get no list. Many of the tests had been written to stop early when the search part gave no answer, so a fix that stepped aside on any doubt passed them. They asserted that the wrong answer was gone. No test checked that an everyday search still got a list.

After one more round the task-writer re-ran the reviewer’s comparison: 5 of 72, and each of the five either matches no listing or names a place no listing holds. The reviewer did not read that round. The tests read 253 of 254 before and after. They could not tell the two versions apart.

3. Give the reviewer the result, not the worker’s account

the reviewer alonethe reviewer alone: 2929worker and reviewerworker and reviewer: 99the worker alonethe worker alone: 88the task-writer's extra checkthe task-writer's extra check: 44the test written firstthe test written first: 11run by the workerthe human ownerthe human owner: 00
Who caught each of the 51 recorded defects in 13 runs. A row can group several findings of one kind from one run, so these are rows of a table, not a count of every flaw. The reviewer was not given the worker's report.

The reviewer alone caught 29 of the 51 rows, and shared 9 more with the worker. Its first answer was no in 8 of the 13 runs. The test written first is credited with one row, and the worker was the one who ran it. The person stepped in zero times.

One habit of the reviewer found a class of error no test covered. Nine rows are wrong facts in texts, and the reviewer caught seven of them. No test case tested what a summary sentence claimed: the cases held numbers to the record and stopped there. In run 13, the trial’s last, every number in the result was right and three sentences were still false. One carried a cost figure onto a different set of messages. Another said a variant got its original’s “answer” back where only its points repeated. The reviews that found these traced or recomputed every number. Reading each summary line against the numbers beside it is now a written rule.

Three cautions about the count. “Blocking” is the reviewer’s own label, so the fact that all 14 blocking findings came from the reviewer proves nothing. The reviewer was not always right: in run 1 its own formula was off by one, and the worker caught it. And no run gave a reviewer the worker’s report, so the record cannot say how much the withholding itself found. In run 6 the reviewer did see the graders’ notes before its own reading, and said so.

So what

What we changed. Four rules on 11 November. The main two: a reviewer reads the task and the test before any work starts, and every test case is shown failing on a broken version of the thing it guards before it is frozen. In the first run under those rules, the task-writer had proved its test against broken versions of its own choosing, 14 of 14. The reviewer removed twelve other rules from a reference solution and the test still passed 14 of 14. The test was rewritten before any work began. The worker then passed it on the first round, and the review of the result had no blocking finding. That is one run, not a rate.

The chat-bot run in section 2 came after these rules. A second agent had reproduced each of its findings, as that run’s plan required. But its tests went to the workers a minute after they were frozen, and no review of the tests before the work is on record. They were wrong in a way the trial had not shown.

What you can copy without this set-up.

  • Have someone other than the test’s author try to break the test. A proof built from the author’s own wrong versions measures the author’s imagination.
  • Before trusting a green suite on a fix, run the old version against the new on ordinary inputs and count what changed.
  • Read every summary sentence against the numbers beside it. A result can be false with every figure correct.

What this does not show.

  • All three agents are models from one vendor. An error they share leaves no row in the record, and nothing here measures that.
  • Nothing was done without the loop, so there is no claim that it produces better work than one session would.
  • There is no cost figure. The thirteen runs took a median of 72 minutes each, from the commit that froze the test to the commit that closed the run.
  • The record was written by the task-writer, the role being measured. When it was compiled it contradicted itself in five places.
  • In the chat-bot run, the other set of tests the workers fixed against stood at 330 of 451 after the fix. Some of the rest fail because the fix replaced the code they were anchored on; the others cover work left for later runs.

The record

The field note has the thirteen runs, the 51 defect rows, the seven wrong tests, the chat-bot run’s figures and count.py, which recomputes the trial’s counts. The briefs, tests and reviews themselves stay private: they quote private code. Before publication a second agent read the note, and then this post, sentence by sentence against the private record. It raised 16 objections to the note and 14 to the post, most of them to sentences that said more than the record did. Each was corrected.