In June, after the US government suspended two Claude models for every non-US user overnight, we wrote about what CasaSol experienced: nothing. No API call to interrupt, no data in transit to retain, no dependency to lose. We closed that post with a promise — Chronos experiment 018 would stop asserting that and start testing it. Three scenarios: stop the inference daemon, delete the model weights, cut the network entirely. The hypothesis was that all three degrade gracefully with zero data loss and configuration-only recovery.
Two of the three came back exactly as predicted. The third one made us reboot the machine.
Stopping Ollama was the cleanest result of the day, and it corrected something we’d assumed rather than checked. The pre-registered hypothesis expected CasaSol’s search tools to degrade when the inference daemon went down. They didn’t degrade at all — not because the architecture is unusually resilient, but because the retrieval layer was never coupled to Ollama in the first place. The embedding model behind property search is a local sentence-transformers model, and its own docstring says “no Ollama needed.” We checked before assuming, the way exp_020 taught us to, and confirmed it directly: search_properties returned real listings before, during, and after the daemon was down.
The part that is Ollama-dependent — the Telegram bot’s intent routing and response generation — failed exactly the way you’d want a dependency to fail. The router hit a connection error immediately; the response generator retried twice over about three seconds, then returned its built-in fallback message. No stack trace, no crash, nothing a user would read as broken beyond “try again in a moment.” Restarting the daemon brought everything back within about a second. One command, and the corpus — checked directly against SQLite, bypassing Ollama and the vector store entirely — held its exact row count throughout.
Deleting the model weights told the same story with a sharper edge. We removed the 17GB response model but left the smaller routing model untouched, so intent classification kept working right through the outage — only the final answer generation failed, with an unambiguous “model not found” error. Same graceful fallback. Recovery was a single ollama pull, which took about four and a half minutes for the re-download, after which the bot answered normally again. Zero data loss, again confirmed directly against the database rather than assumed from the absence of an error.
Both of these are exactly the kind of boring, confirmed-safe result Chronos is for. Not everything worth testing is worth a dramatic finding.
Cutting the network is where the boring streak ended, and where the real finding is — though not the one we set out to find.
The pre-registered test assumes an operator can still check on the machine locally while its network is down: confirm the database is readable, re-run a query, watch the local inference layer keep working while only the outbound-dependent pieces fail. We wrote that assumption into the hypothesis without noticing it was an assumption. The machine that hosts CasaSol’s inference has no physical console workflow at all — every session that reaches it, including the one running this test, gets there over SSH from a second machine. There is no “local” check available that doesn’t itself depend on the network we were about to cut.
We armed a safety net before applying the block: a 90-second background timer set to undo it automatically. Then we ran pfctl to block all outbound traffic, and the SSH session carrying the test died in the same second. The safety net didn’t fire — it was a plain backgrounded shell job, not detached from the session that had just been severed, so it almost certainly died with it rather than surviving to do its job. Five minutes of no reconnect later, the only path back was a full reboot of the machine.
Everything came back clean. The inference daemon restarted itself on boot with no manual intervention. The database held its exact row count from before the incident. Zero data lost, in the sense the pre-registration actually cared about. But the specific question we set out to answer — does inference keep quietly working while the network is cut — was never actually observable, because the same block that correctly isolates the machine from the outside world also isolates the only channel we had for checking on it.
That’s a real distinction, not a technicality: sovereignty of inference and sovereignty of observability are not the same property. CasaSol’s corpus and model plausibly kept working through that outage exactly as designed — we have no reason to think otherwise, and the pre- and post-incident state matches perfectly. But “plausibly kept working, we couldn’t check” is a materially weaker claim than “confirmed working,” and the gap between those two is itself a finding worth having, not a footnote to bury.
Two confirmed, one honestly inconclusive, and one unplanned reboot of a production machine. We could have written this up as “2 out of 3 passed” and left it there. We’re writing it up as what it actually was, including the part where our own test design had a blind spot that a real incident was needed to surface — because that’s the same discipline the last hardening post was built on: a claim about what a system does is not evidence until you’ve run it and watched what happened, and sometimes what happens is that you learn your test had an assumption baked in you didn’t know you’d made.
The fix here isn’t a better prompt or even better code — it’s a different kind of safety net next time: one that survives the exact failure it’s meant to protect against, and a test design that accounts for the fact that on this machine, checking on the patient and keeping the patient alive run through the same wire.
The full experiment — pre-registered hypotheses, raw evidence for all three scenarios, and the incident writeup — is Chronos experiment 018: exp_018_sovereignty_resilience.