Nous Research's MoE field notes: what actually happens when 1 trillion parameters hit 1024 GPUs

Nous Research published a detailed technical report on scaling Mixture-of-Experts (MoE) expert parallelism for large-scale pretraining using DeepEP, an open-source library for efficient expert communication. The report draws from the team's experience operating a multi-terawatt-month cluster to pretrain an unnamed 1-trillion-parameter MoE model. The document goes beyond benchmark numbers into implementation realities: which configurations worked, which bottlenecks appeared, and how the team resolved them.

The core challenge with MoE at scale is expert load imbalance. When routers send disproportionate traffic to certain experts, some GPUs sit idle while others queue. DeepEP addresses this through two-level parallelism: data parallelism across experts, all-reduce across shared experts, combined with dynamic routing that re-balances load during training. The Nous report confirms that DeepEP achieves near-perfect load balance within 2% deviation at 64 experts distributed across 1024 GPUs. For teams building their own clusters, this kind of operational detail is often more instructive than benchmark scores, as agent-first environments demonstrate.

Key numbers from the report: throughput of 380 t/s/gpu on H100 for a 1T MoE model with 64 experts; expert communication overhead at 3% of total training time after DeepEP optimizations; load imbalance without DeepEP at 34% idle time on the worst GPU; load imbalance with DeepEP under 2% idle time. These figures matter because they quantify the gap between a prototype and a production system, a gap that remains stubbornly human despite progress in AI infrastructure, a pattern explored in the discussion of vibe coding's limits.

The report contrasts DeepEP with conventional all-to-all expert communication, which becomes a bandwidth bottleneck at scale. DeepEP uses a custom communication scheme that overlaps expert computation with data exchange, hiding communication latency behind compute. The Nous team also describes their production cluster topography: a Slurm-managed 1024-node cluster with H100 GPUs connected via NVLink and InfiniBand. Knowing which hardware and scheduler combos work together is a practical concern that rarely makes it into published architecture papers, much like the infrastructure insights shared in Ollama's $88 million bet on open-weight models.

The practical implications for other labs: DeepEP is not a drop-in solution. The Nous report warns that expert parallelism gains are sensitive to model architecture factors like number of experts, capacity factor, and top-k value. The ideal configuration must be determined empirically per model family. The team releases their configuration files and monitoring scripts as reference templates. This kind of hands-on guidance is reminiscent of how groups tackling hard infrastructure problems, whether in cloud GPU hopping or other distributed training challenges, share templates rather than just results.

In the broader landscape, Nous Research's MoE work joins contributions from Google with GLaM, Meta with V-MoE, and Mistral with Mixtral. But where those labs publish final architecture papers, Nous's field notes emphasize the operations layer: what the cluster admin needs to know to keep training stable over multi-month runs. That perspective is rare in open research and directly useful for any lab building a large-scale MoE cluster, much like the practical orientation found in Alibaba Cloud's Qoder approach to codebase visibility.