A relay race of rented GPUs trained NanoColibri's 2.7B MoE for $200

Pretraining a language model from scratch usually means a cluster, a six-figure compute bill, and people paid to keep the run alive. The answer from a new open-source project comes to about $180 to $260. NanoColibri-Instruct, a 2.7B-parameter Mixture-of-Experts model with 0.34B active parameters per token, went from random weights to a usable model for that price, no cluster, contributors taking turns on rented GPUs one at a time. Code, checkpoints, and a public ledger of who trained what live in the training repo, one more data point for the open-weight economics that carried Gemma 4 to 300 million downloads.

The model was never the end goal. The build is the story, as it was for Grok 4.5's release. The writeup frames Nano as the proof-of-loop for a larger idea: models whose int4 containers are deliberately bigger than a consumer machine's memory, streaming experts from NVMe under a fixed RAM budget. A 16 GB laptop cannot hold a 24 to 28B model in RAM, a constraint Unsloth attacked from the other direction when it squeezed the 975B Inkling into 270 GB for a desktop. In a fine-grained MoE, each token touches only the dense backbone, one always-resident shared expert, and a couple of 3 to 4 MB routed experts. Cache experts with an LRU, let the disk serve the misses. Nano exercises the whole pipeline, int4 export included, at a price where mistakes don't hurt.

The architecture matches the serving engine's native form, HYV3: 24 layers (one dense, 23 MoE), hidden width 1024, 64 experts of 512 width with top-2 routing, a fat 2048-wide shared expert that never leaves memory, 4x grouped-query attention with per-head QK-norm, and a sigmoid-plus-bias router. Load balancing follows the DeepSeek-V3 recipe: nudge each layer's expert bias by gamma times the sign of target minus load.

Training by relay: a compare-and-swap on the Hub

You cannot split sequential pretraining across volunteers the way you split a codebase. Every step depends on the previous one, so the run lives in a single Hugging Face model repo, weights plus optimizer state, training_state.json, RELAY.json, LEDGER.md, and contributors take turns: claim, pull, train a leg, push, release.

The collision is the part that has to be airtight. If two people train the same leg, whoever pushes second silently destroys a GPU-day. So the baton claim is a lease, a real compare-and-swap against the Hub rather than a polite convention, renewed by heartbeat while training, which means a preempted spot instance frees the baton on its own. A read-only dashboard on the training box shows loss, throughput, expert balance, and who holds the baton.

It worked: 20,000 updates, roughly 5.2B tokens of FineWeb-Edu, then 1,500 updates of chat SFT on smol-smoltalk, about 90 H100-hours all in.

One result is worth stealing: train with top-2 routing even if you will serve with top-1. At top_k=1 the routed weight collapses to a constant and the router gets essentially zero gradient from the LM loss, so routing stays frozen at its random init. top_k=2 restores gradient flow through competition between two winners. At serve time the engine still runs one expert per token. The aux-loss-free balancing held at this scale: dead-expert fraction stayed low, load stayed healthy, and the mild routing skew that remains is what a streaming cache likes.

Expert imbalance is the same failure mode at the other end of the MoE spectrum. Nous Research's field notes on pretraining a 1-trillion-parameter model describe the core challenge as routers sending disproportionate traffic to certain experts, leaving some GPUs idle while others queue. The NanoColibri version of that problem cost a few hundred dollars to meet.

What 5.4B tokens buys

The numbers come with methodology attached. Benchmarks ran through an in-repo, lm-evaluation-harness-compatible harness, same prompts, same scoring, and the team ran the dense references through the same code instead of trusting published rows. Rankings live or die by that discipline, whether it is the seven tasks in the table below or the Chatbot Arena's five million votes.

model	active params	tokens	lambada	piqa	wino	arc-e	arc-c	obqa	hswag
NanoColibri (chat)	341M	5.4B	26.3	62.7	49.2	43.4	22.8	22.0	31.1
Pythia-410M @ step3000	405M	6.3B	26.3	59.8	50.9	41.3	18.8	15.6	27.0
Cerebras-GPT-256M (final)	256M	5.1B	29.3	61.3	51.1	41.0	17.0	15.8	27.4

NanoColibri wins 5 of 7 against both baselines, ties LAMBADA against Pythia, and sits within noise on WinoGrande (±1.4 at n=1267). The caveats are public. The Pythia checkpoint is only about 2% into its LR schedule, which handicaps it; the fully annealed Cerebras-GPT-256M row is the conservative comparison, and Nano wins that one too. The LAMBADA lag looks at least partly like a data artifact: FineWeb-Edu contains essentially no fiction, and LAMBADA is built from novels.

The project's reading is measured: at this budget, a 2.7B-total MoE behaves like a good dense model in the 300 to 600M class, the total parameter count buying a real edge over two token-matched dense baselines. The gap to SmolLM2-class numbers is three orders of magnitude of data, not architecture.

Two embarrassing lessons

The first is the page-cache trap. To 'measure' streaming, the team gave the serving engine sub-container RAM budgets on a rented server and got suspiciously flat speeds down to a budget five times smaller than the container. The server's RAM dwarfed the 1.2 GB container, so the OS page cache silently backed every miss. Those runs never touched disk; they characterized the engine's cache bookkeeping. True disk-pressure numbers need memory-capped cgroups on hardware where the cap is enforced and, the team admits, a bigger container. A bench bug found in the speed harness since then is why the post carries zero tokens/s claims. Measurement discipline is cheaper to learn at $200, the kind of quiet correction that passes for this week's actual news.

The second is the spent cosine. A cosine LR schedule fixed to a step target is spent by the time you reach it, with no headroom to keep training on momentum. The successor runs use WSD, warmup-stable-decay: a long stable phase that suits relay legs, decaying once, when the team chooses.

What's next: a 7B rehearsal, then a 24 to 28B container

The roadmap in NEXT_MODEL.md has two stages. Colibri-Micro: 7B total, 1B active, 100B tokens, a code-heavy mix, SFT then RLVR for code, and a container actually big enough, 3.8 GB in int4, to measure real disk streaming on target laptops. Then Colibri-Grande, 24 to 28B total with about 2.4B active, whose container deliberately oversubscribes a 16 GB laptop's memory. Grande's expert count freezes only after Micro's measured miss and speed curves exist. Everything ships open: weights, data recipe, training code, relay protocol, benchmark JSONs, the ledger, following the full-transparency precedent of Ai2's Olmo 3.

Nano was paid for out of pocket. The next stage needs a few thousand H100-hours, roughly $9,000 to $12,000 at market rates, and the author is applying for compute grants. The $200 proof-of-loop is done. The lessons it priced, the page cache and the spent cosine, are exactly the ones a $9,000 run cannot afford to learn fresh.