We Stopped the Kolibri Benchmark Before Scoring One Answer

On Monday our own gate refused to certify our MLX port of Aleph Alpha’s Kolibri-1, and we published that instead of scores. Taking the failure apart left us two lessons: Force the routing. Kolibri sends each token to 6 of 384 experts in every layer. Two runs of our port that differed only in prefill chunk size picked different experts and drifted apart over a long context. Logits have to be compared with the expert choices forced. Update the runtime. The batched failure on our M5 Max was an MLX bug, already fixed in MLX 0.32.3. exp_037 is the experiment that post promised: the same questions and hypotheses, and most of the kit, behind a gate rebuilt on those lessons. We froze it on Wednesday, 7 October, before any gate run, with no amendment type that can change a threshold or a verdict rule. It ended this morning: ...

October 9, 2026 · 28 min · Nestor

We Ported Aleph Alpha's Kolibri to a MacBook. Our Own Gate Said No.

Updates: 5 October: we narrowed the M5 result to one MLX operation and reported it. 6 October: it turned out to be a known bug, already fixed in MLX 0.32.3, a newer release than the one we ran. See the end of section 3. Aleph Alpha released Kolibri-1 on Saturday, 3 October. It is a mixture-of-experts model: 78.1 billion parameters in total, 3.46 billion active per token, trained for English and German and published under Apache-2.0. That day it had no MLX build and no GGUF, and no local runtime supported it. We wanted to know three things: ...

October 5, 2026 · 22 min · Nestor