Local First AI

In exp_035 we fine-tuned Qwen3-4B three times on answers written by a much stronger model. It learned the shape of a good answer, and lost substance every time. The best build we had was the untrained model behind deterministic code we call the glue. Then Coinbase described post-training a 9B model with GRPO against a reward computed from real outcomes, and beating a frontier model on that task. That suggested the cheap experiment our post had asked someone to run: change only the training objective. Our model would write its own answers, code would score them, and no stronger model’s text would appear anywhere in training. Same base, same retrieval, same prompts. ...

On Tuesday Andrei asked whether we should train Kolibri-1, Aleph Alpha’s English–German mixture-of-experts model, as his property agent. We advised against training. In exp_035, fine-tuning a small model on our study notes lowered its use of the retrieved facts in all three runs. The best build was an untrained model behind deterministic glue. So exp_038 asks a narrow question. Put Kolibri into that glue as the answering model, hold everything else fixed, and see if it beats gemma4:26b, the model CasaSol already runs. German was the reason to try: many buyers on the Costa del Sol are German, and Kolibri was trained for German. ...

On Monday our own gate refused to certify our MLX port of Aleph Alpha’s Kolibri-1, and we published that instead of scores. Taking the failure apart left us two lessons: Force the routing. Kolibri sends each token to 6 of 384 experts in every layer. Two runs of our port that differed only in prefill chunk size picked different experts and drifted apart over a long context. Logits have to be compared with the expert choices forced. Update the runtime. The batched failure on our M5 Max was an MLX bug, already fixed in MLX 0.32.3. exp_037 is the experiment that post promised: the same questions and hypotheses, and most of the kit, behind a gate rebuilt on those lessons. We froze it on Wednesday, 7 October, before any gate run, with no amendment type that can change a threshold or a verdict rule. It ended this morning: ...

Updates: 5 October: we narrowed the M5 result to one MLX operation and reported it. 6 October: it turned out to be a known bug, already fixed in MLX 0.32.3, a newer release than the one we ran. See the end of section 3. Aleph Alpha released Kolibri-1 on Saturday, 3 October. It is a mixture-of-experts model: 78.1 billion parameters in total, 3.46 billion active per token, trained for English and German and published under Apache-2.0. That day it had no MLX build and no GGUF, and no local runtime supported it. We wanted to know three things: ...

On Tuesday evening one of us sat in the audience at 42 Málaga, the coding school Fundación Telefónica runs in the city’s digital-content hub on Avenida Sor Teresa Prat, and watched five people, in four talks, take apart a question most of us skip. How do you know an AI’s answer is any good? Malaga-AI is a local community that has been running meetups for three years. Each year it runs a certified study group, and this year’s, from May to July, was on AI safety and evaluation, or in the programme’s words “trust, fairness and alignment”. Each student picked a question, built or borrowed a test, and ran it against real models. Tuesday was the final showcase. ...

The idea fit on an index card. A small model on an iPhone answers questions about buying property in Spain — in English, Polish or Spanish — the way a colegiado would: gives the Spanish term, cites the tema and the article, and refuses to make the decision for you. The facts never live in the model. They come from retrieval over a study guide one of us wrote this summer while qualifying as a real-estate agent, about 300 KB of it, on the device. Only the manner goes into the weights: which language to answer in, plain text, cite, hand off. Facts in retrieval, form in the weights. Nothing leaves the phone. ...

A while back we audited the machine that runs our local AI: the same Mac Mini that does all of the inference, the one whose entire selling point is that your data never leaves it. An earlier pass (Hardening the Inference Node) went looking for the dramatic stuff, what someone could reach with elevated privileges. This pass asked a smaller, meaner question. What’s readable with no privileges at all, just ordinary code running as the everyday account that already runs the AI, all day, by design? ...

We were comparing AI models this week — the one we already rely on (it’s called gemma4:26b) against a newer one just released (qwen3.8:27b) — and a third model already sitting on the machine, gemma4:31b, looked completely dead in the middle of it. Every request to it just hung. Not slow. Nothing came back at all, for ten minutes at a time. The easy conclusion, the one we almost wrote down, was that the model didn’t work on our hardware. Some models just don’t run well on some machines. Fine, move on. ...

In June, after the US government suspended two Claude models for every non-US user overnight, we wrote about what CasaSol experienced: nothing. No API call to interrupt, no data in transit to retain, no dependency to lose. We closed that post with a promise — Chronos experiment 018 would stop asserting that and start testing it. Three scenarios: stop the inference daemon, delete the model weights, cut the network entirely. The hypothesis was that all three degrade gracefully with zero data loss and configuration-only recovery. ...

CasaSol Guide is a Telegram bot backed by gemma4:26b: a property advisor for the Costa del Sol that answers questions using a retrieval corpus of listings its human operator has personally visited, a curated area guide, and — as of last week — a community contribution channel called /witness, where any invited beta user can submit a first-hand observation about a neighbourhood. Once an admin approves it, that observation gets embedded and joins the same knowledge base the bot draws on for everyone. ...