Muon groks modular addition faster, then its solutions collapse

The paper, posted to arXiv on August 7, 2026, works on a deliberately small stage: a transformer learning modular addition, one of the standard test beds in grokking research. Grokking is the delayed-generalization effect where a network memorizes its training set for a long stretch, then suddenly snaps into a generalizing solution. In the split the authors treat as standard, Muon takes the hidden matrices and AdamW takes the embeddings and output head. Muon reaches the generalizing state faster. Then it loses it, in every one of the nine configurations they tested.

The core result: Muon groks faster, then falls apart

All nine configurations on (a+b) mod 113 grok and later drop below the generalization threshold. The instability is not a fluke of a single setup. It persists across two moduli, two widths, two training fractions, the subtraction variant, and different depths. The AdamW baseline is not immune either: across five seeds it falls below threshold on four, bottoming out at 27.59%.

A few numbers give the shape of the problem.

The headline is that a training run which solves the training set, and then generalizes, can still fall apart afterwards, with no new data in sight. That is the puzzle the paper sets out to explain, and it belongs to a known family of failure: training that quietly undoes itself, a problem RDPO set out to fix in multi-task reinforcement learning, where reward signals that drive alignment often work against each other.

Where the failure lives: the representation-readout interface

The authors trace the collapse to the interface between the learned representations and the readout layer. One technical detail matters: that interface can only be identified jointly, up to an invertible map that the loss does not select. The training loss simply does not care which member of that equivalence class the network lands on.

After the training set is solved, the gradient falls to around 1e-6. That is where the two optimizers part ways. Step-size elasticity is -0.03 for Muon against +1.5 for AdamW, and the Muon group moves 8.0 times faster per parameter. A near-zero gradient is not a quiet place for Muon. Its updates still have room to do damage, the exact opposite of the gradient gap SoftReason bridges so neural networks can reason.

Freezing one group removes the failure

The causal experiments are the cleanest part of the paper. From bit-identical starting states, freezing either group prevents the failure. Freezing embeddings and the readout removes it across five runs spanning 451,400 post-grokking steps and five paired seeds. The unfrozen arms record 137 to 321 sub-threshold evaluations. The frozen arms record none.

Stripping out Muon's normalization and orthogonalization is not a fix. Without it, the representation collapses from 326 effective conjugate pairs to 4, shows no recurrent collapse, and fails terminally. The instability is bound up in Muon's own update machinery, not in something generic about modular arithmetic.

Fourier analysis: the circuit survives and gets outvoted

Fourier filtering separates two distinct failure modes, an interpretive step in the same spirit as saliency maps in explainable-RL research. Across 43 checkpoints, five seeds, and three regimes, the task-aligned family reaches exactly 100% accuracy when evaluated on its own. In circuit failure, that family no longer solves the task. In masking, it stays perfect while the full model sits at 45.85%. The aligned circuit keeps a positive margin on every example, including the errors. It is simply outvoted by a near-equal adversarial remainder. Rescaling the family restores 99.9%.

The authors read grokking as the same condition resolving in the other direction. Structural details back this up: the task selects the family, swapping (k,k) for (k,-k) under subtraction, and across an abrupt collapse the standard Fourier support is unchanged, with the power-distribution cosine at 0.9899. The computation is still there. What changes is how much the readout lets it count.

What this means for optimizer benchmarks

The obvious takeaway is that grokking speed is a weak success metric on its own. A run that reaches the generalizing state first is not clearly better if it cannot hold that state. Optimizer comparisons that stop at the grokking curve are measuring the first half of the story.

Caveats first. This is one task family, modular arithmetic, at small scale, and the paper is a preprint. Fresh arXiv results earn a read, not a verdict, as in our read of a 33-page preprint five days after it appeared. Nothing in the abstract claims the effect carries over to large-scale training. What it does offer is a concrete habit for the next benchmark you run: check post-grokking stability, not just time-to-grok.

On these small tasks, the difference between the two optimizers is not who learns first. It is who is still right once the loss has gone quiet.