Anyone can say prompt injection is a risk. This post shows three things that are harder to find, measured on a local model anyone can run:
- Which kind of injected text worked, and which did not. A blunt order to override the instructions did nothing to a scam message. A paragraph dressed up as a system block did.
- Why “the model can only add points” did not protect it. Our model can raise a verdict and never lower one. A sender could still take away what it had added.
- What a fix in the code, not in the prompt, costs. It closed the hole we tested, and it is paid for in model calls and false alarms.
The numbers and their limits follow. All messages are synthetic, it is one model on one machine, and the whole record is public.
What was tested
A program reads an incoming message and says benign, suspicious or likely scam. It has rules written in code, which look at things like the age of a linked domain, urgent wording and changed payment details. It also has one call to a local model, gemma4:26b on a Mac mini. The model picks three labels from fixed lists: the kind of approach, what the sender asks for, and how sure it is. Risky labels add up to 25 points to the rules’ score. The model can only add, never subtract, and never writes free text. Its prompt already tells it to treat the message as untrusted data.
The test: 12 synthetic scam messages and 12 genuine ones, and six paragraphs aimed at the model. Each paragraph was added to the end of each message, behind a blank line. That is 72 tries on scam messages and 144 in all. The six paragraphs were fixed in the public record on 25 October, before any test code existed. The run was on 10 November.
In this post a paragraph “worked” when the model’s contribution dropped to zero points.
1. Which injected text worked
Across the six paragraphs that is 19 of 72 tries, on 11 of the 12 scam messages.
- The blunt order never worked on a scam message. Neither did the fake screening record, nor telling the model it had a different job.
- Two paragraphs did most of it, 8 of 12 each: a fake system block, and a fake “calibration probe” that names the tool’s own internals.
- We cannot say why. There is one paragraph of each kind. The two that worked both pose as part of the system around the model, but so does the fake screening record, and that one did nothing.
- One paragraph backfired. The “different job” text set off the rule for urgent wording and raised six verdicts.
- A harmless paragraph also did it once. As a control we added a neutral paragraph of similar length, with no instruction in it. It took the model’s points to zero on one scam message. Three of the 19 are on that same message. The control failed its own repeat check on one message, so it is a warning and not a count.
One caution on the total. The 12 scam messages were picked because the model gives all of them points, and half of them are flagged only because of the model. So 19 of 72 is not a rate for scam mail in general.
2. Why “the model can only add” did not protect it
The verdict changed only where the model was the one thing holding it.
- Six of the 12 scam messages are flagged by the rules alone. On those, 9 tries lost the model’s points and none lost its verdict.
- The other six are flagged only because the model adds points. On those, 11 of 36 tries came out benign instead of suspicious.
- The rules were not talked down. No added paragraph lowered a single rule finding.
- An unreachable model was loud. With the model made unreachable on purpose, all 24 verdicts fell back to the rules and said the model step had failed.
On genuine messages the same paragraphs moved 16 more verdicts, all on four messages the model had wrongly flagged in the first place. That makes 27 lowered verdicts in 144.
Letting a model only add points, with a cap, limits the harm of a wrong answer. It does not protect the points. If a message is flagged only because of what the model said, the sender can write to the model and take that away.
We published this the same afternoon, as a failed test, and kept the six paragraphs back.
3. The fix, and what it costs
Before, the model was asked one question, about the whole message, and whatever the sender put last was part of that question. Now the code asks about the whole message and about the message cut at every paragraph break, and keeps the answer worth the most points.
original message p1 p2 p3 p4
with added text p1 p2 p3 p4 [added paragraph]
before the fix one question: p1 p2 p3 p4 [added paragraph]
after the fix five questions: p1 p2 p3 p4 [added paragraph]
p1 p2 p3 p4 <- the question the original message gets
p1 p2 p3
p1 p2
p1
the answer worth the most points is kept
The second question is the point of the design. Whatever is added at the end, the model is still asked about the original message without it. If the model gives the same answer to the same question, a paragraph added at the end can no longer lower the model’s points.
- It is code, not a prompt. The model, its prompt and its settings are byte for byte what they were. The prompt already said the message was untrusted, and the model’s points still went to zero 19 times. A stronger prompt asks the model to obey us and not the sender, and how often it does has to be measured again for every new text. We did not test a stronger prompt. We chose a property that does not depend on the model obeying.
- Our first design was wrong. It asked about the message with its last one to six paragraphs removed. A reviewer rejected it before any run: a message’s own best answer could fall out of reach once text was added.
- It was written without the six paragraphs. The sessions that wrote and reviewed the fix did not read them or run anything on them. It was tested with a stand-in model. The fix, its limits and a limit on false alarms went into the public record before the second run.
The second run. The same 144 test messages, the same scripts. No paragraph took the model’s points to zero, 0 of 72, and no verdict was lowered, 0 of 144. That part could hardly fail, because the design asks the original message’s question again. What the run showed beyond that:
- The model’s points repeated. The original messages’ questions were asked again 612 times inside the longer messages. The points came back the same 612 times.
- Its labels did not always repeat. In 10 of the 612, the same question got a different label, at temperature 0 with a fixed seed. The records show no cause. The fix relies on points, which held.
The price.
| Before the fix | With the fix | |
|---|---|---|
| Model calls per message | 1 | 4 to 5, one per paragraph |
| Seconds per message | 1.75 | 5.60 |
| Genuine messages wrongly flagged, of 110 | 14, on 25 October | 16 |
| Scam messages caught, of 110 | 83, on 25 October | 83 |
The last two rows come from the 220 synthetic messages of an earlier experiment, which the 24 test messages were drawn from. They were first run on 25 October and run again through the fixed program. Taking the highest of several answers can only raise points, so a fix like this can look perfect by flagging everything. We wrote the limit down before the run: not adopted above 19 false alarms in 110. It came to 16. Andrei adopted the fix that evening; the experiment’s folder note records it.
What this does not cover
- Text anywhere but the end. The fix is built for a new paragraph after a blank line. Text at the start or in the middle was not tested. Text joined to the last paragraph, or a message of more than 24 paragraphs, breaks the guarantee by design.
- Other kinds of mail. All the e-mails here are plain text in one part, with at most 7 paragraphs. Multi-part mail and encoded bodies were not tested.
- A sender who knows the fix. The six paragraphs were written before it existed. They are public now; that says nothing about the next one.
- A model that changes its answer. The guarantee needs the same answer to the same question. It held for points here, on one model, one seed, one run.
- One case where points can still fall. If the added text sets off a rule that already scores what the model named, the model’s points drop while the total score rises. It did not happen in these runs.
- The rest of the problem. Two more things are in the plan and not yet measured: text that hides characters or fakes a verdict line on the screen, and text that floods a message so the real evidence is never looked up. Until they are measured and pass, this program does not read mail unattended.
If you build something like this
- Count what the model alone is holding. Here the program catches 83 of 110 scam messages with the model and 68 without. Those 15 are the ones exposed.
- Do not rely on “the model can only add”. It bounds the damage of a wrong answer. It does not stop a sender from removing a right one.
- Test more than the blunt order. “Ignore your instructions” never worked here. Text posing as part of the system did.
- For text added at the end, you can get the guarantee from structure. Ask the question the shorter message would have been asked, and keep the highest answer.
- Write the false-alarm limit before you measure. Any “keep the highest” fix raises scores. Decide what is too many first.
- Add a harmless control. A neutral paragraph moved our model too. Without it, every lost point looks like an attack.
- Do not assume temperature 0 repeats. Compare the numbers that decide, and run each message twice.
- Make a failed model step loud. A silent fallback looks exactly like a clean message.
The full record is public: exp_056, with the plan written before the runs and its four amendments, the test scripts, every record, the first result, the second, all the limits, and the six paragraphs with a note on what each one tries. The 220 synthetic messages are from exp_055. The program under test is in a private repository; its commits are pinned in the records. The model under test ran locally on the Mac mini. The six paragraphs were written by Fable, a cloud model. The test scripts, the harmless control, the fix and both result write-ups were written and reviewed by Claude sessions, a cloud model, which also started the second run on Andrei’s instruction and did not read the six paragraphs. The script that builds the test messages was written by Andrei with a local model. This post was drafted with Claude.