Local-Llm

Updates: 5 October: we narrowed the M5 result to one MLX operation and reported it. 6 October: it turned out to be a known bug, already fixed in MLX 0.32.3, a newer release than the one we ran. See the end of section 3. Aleph Alpha released Kolibri-1 on Saturday, 3 October. It is a mixture-of-experts model: 78.1 billion parameters in total, 3.46 billion active per token, trained for English and German and published under Apache-2.0. That day it had no MLX build and no GGUF, and no local runtime supported it. We wanted to know three things: ...

The idea fit on an index card. A small model on an iPhone answers questions about buying property in Spain — in English, Polish or Spanish — the way a colegiado would: gives the Spanish term, cites the tema and the article, and refuses to make the decision for you. The facts never live in the model. They come from retrieval over a study guide one of us wrote this summer while qualifying as a real-estate agent, about 300 KB of it, on the device. Only the manner goes into the weights: which language to answer in, plain text, cite, hand off. Facts in retrieval, form in the weights. Nothing leaves the phone. ...

A while back we audited the machine that runs our local AI: the same Mac Mini that does all of the inference, the one whose entire selling point is that your data never leaves it. An earlier pass (Hardening the Inference Node) went looking for the dramatic stuff, what someone could reach with elevated privileges. This pass asked a smaller, meaner question. What’s readable with no privileges at all, just ordinary code running as the everyday account that already runs the AI, all day, by design? ...

We were comparing AI models this week — the one we already rely on (it’s called gemma4:26b) against a newer one just released (qwen3.8:27b) — and a third model already sitting on the machine, gemma4:31b, looked completely dead in the middle of it. Every request to it just hung. Not slow. Nothing came back at all, for ten minutes at a time. The easy conclusion, the one we almost wrote down, was that the model didn’t work on our hardware. Some models just don’t run well on some machines. Fine, move on. ...

In June, after the US government suspended two Claude models for every non-US user overnight, we wrote about what CasaSol experienced: nothing. No API call to interrupt, no data in transit to retain, no dependency to lose. We closed that post with a promise — Chronos experiment 018 would stop asserting that and start testing it. Three scenarios: stop the inference daemon, delete the model weights, cut the network entirely. The hypothesis was that all three degrade gracefully with zero data loss and configuration-only recovery. ...

CasaSol Guide is a Telegram bot backed by gemma4:26b: a property advisor for the Costa del Sol that answers questions using a retrieval corpus of listings its human operator has personally visited, a curated area guide, and — as of last week — a community contribution channel called /witness, where any invited beta user can submit a first-hand observation about a neighbourhood. Once an admin approves it, that observation gets embedded and joins the same knowledge base the bot draws on for everyone. ...

The pitch for local-first AI is simple: your documents never leave your hardware. It’s a true claim, and it’s also an incomplete one, because it quietly assumes the hardware itself is secure. Nobody had actually tested that assumption on the machine doing the work — a Mac Mini M4 Pro that runs local inference for a client-facing document-processing deployment, all day, every day. This is what happened when we did. ...

[miktam — preface] We needed a data sovereignty legal brief — the kind you hand to a lawyer as a starting point. The question: can AI produce something a lawyer won’t immediately dismiss? A single model drafting the document was never going to be sufficient. The same model that writes an overclaim won’t detect it. So Nestor designed an adversarial pipeline: a drafter followed by three panelists with explicitly conflicting mandates. The result — zero of seven claims survived unchanged, and the panel caught two critical issues that would have made a Gibraltar lawyer distrust the document on page one. ...

TL;DR MLX does not cliff through 40K tokens on Mac Mini M4 Pro. MLX prefill at 15K: 1.650 ms/tok. Ollama FA=0 at 15K: 1.774 ms/tok. Difference: 3%. Two independent runtimes. Same hardware. Same conclusion: the ceiling is memory bandwidth, not attention kernel. The Flash Attention cliff from Exp 007 was an Ollama/llama.cpp artefact. Not Apple Silicon. Not unified memory. Not the model. Saw someone running gemma4:26b-mlx directly — not through Ollama, the MLX runtime natively. Left a reply: we hit a context cliff on Ollama that turned out to be a Flash Attention flag issue. Curious if you’ve seen similar behaviour on the MLX backend? ...

TL;DR All three frontier models scored 5/8 net. The local model scored 0/8. Haiku ($0.095) = Sonnet ($0.291) = Opus ($0.611) on this rubric. The cost/quality curve is a single step: $0 (local) → $0.09 (cloud), then flat. Upgrading from Haiku to Opus costs 6.4× more and buys zero additional rubric points. Two items evaded every model. One bonus bug was found only by Sonnet. Two tweets on my timeline last week. @Prathkum (79.7K views): “We don’t need a more powerful model right now. What we need to solve is the cost problem.” @nix_eth: “I don’t think intelligence, capabilities, and cost are all tied together.” ...