The CIFAR-10 ablation that should make every AI researcher rethink their benchmarks

For thirty years, one uncomfortable fact has shadowed deep learning: the algorithm that trains almost every modern neural network is not how brains learn. Backpropagation demands that the backward pass reuse the exact transpose of the forward weights, the weight transport problem Francis Crick flagged in 1989, and no known biological mechanism ships weights backward like that. A stream of alternatives has tried to close the gap. Almost all of them stall past MNIST.

A new paper from Sakana AI, authored by Yutaro Yamada, Luca Grillotti, Rujikorn Charakorn, Sebastian Risi, David Ha, and Robert Tjarko Lange, takes a rule called Error Diffusion further than it has gone before, onto color images and continuous-control robotics. The headline result is that it works at all under the strictest biological constraint. The more interesting result sits in the ablation tables, and it's a warning for the entire field. This isn't Sakana's first attempt at building more flexible AI systems, they've also worked on routing models to tasks instead of forcing one model to do everything, as seen with their Fugu routing system.

What's in this report

• Why backprop breaks biological plausibility, and what Dale's principle adds
• The dual-stream trick: two neuron populations, four weight matrices, one global error
• The scorecard: 96.7% on MNIST, 61.7% on CIFAR-10
• The ablation reversal that exposes single-benchmark evaluation
• Reinforcement learning: where Dale's tax gets cheaper
• What it costs, and why hardware people should care

Dale's principle: the rule most AI ignores

Biological neurons obey Dale's principle: a given neuron is either excitatory or inhibitory at all of its synapses, it does not flip sign connection by connection. Artificial networks ignore this entirely, letting any weight go positive or negative at will. That freedom makes backpropagation efficient, and it is also what makes it biologically fictional.

Enforcing Dale's principle is not cosmetic. It forces a network to coordinate separate excitatory and inhibitory populations, which fundamentally changes how a network can assign credit during learning. Prior backprop-free methods, Feedback Alignment, Direct Feedback Alignment (DFA), predictive coding, all dodge weight transport, but every one of them still allows arbitrary-sign weights. None are Dalean. For a broader look at how different training approaches stack up, the cascade distillation method from Mistral offers a completely different angle on efficient learning.

The dual-stream architecture

Error Diffusion, originally proposed by Kaneko in 2000, is a local rule: a weight update depends only on presynaptic activity, a postsynaptic activation derivative, and a single global error sign. Sakana's team makes it Dale-compliant by splitting every layer into a positive (excitatory) stream and a negative (inhibitory) stream, with four non-negative weight matrices per layer. Cross-stream connections are hardcoded to subtract, so inhibition is structural while every learnable parameter stays non-negative.

The cost is immediate: roughly 4x more parameters than an unconstrained network of the same width (about 32M versus 8M for a DFA baseline). To reach beyond the binary classification the rule was born for, the authors add modulo error routing, each hidden unit is assigned a fixed output channel by i mod C, and receives that channel's error as its learning signal. No transposed weights, no random feedback matrices, just a deterministic map from hidden units to outputs.

The scorecard

With three classification-specific tricks, layer-specific sigmoid widths, batch-centered class error, and asymmetric excitatory/inhibitory initialization, the full model reaches 96.7% ± 0.1 on MNIST and establishes a 61.7% ± 0.7 baseline on CIFAR-10. The paper is candid: ~62% on CIFAR-10 is nowhere near competitive with ordinary gradient methods. But it is the first time Error Diffusion has been applied to convolutional networks at all. A prior 2026 study by Fujita managed ~55.2% only with a flattened MLP.

The unconstrained DFA baseline scores higher, 97.6% and 69.1%, but it does so with roughly 2.84M negative weights, violating Dale's principle outright. And the gap tells its own story: 0.9 points on MNIST, widening to 7.4 points on CIFAR-10. The tax on biological honesty grows with task difficulty. This kind of nuanced performance gap is reminiscent of what Mistral's OCR 4 team found when they looked past their own benchmark scores to the real-world audit.

The reversal that matters

Here is the finding that reaches past this one architecture. When the authors remove each of the three classification tricks in turn, the importance hierarchy does not merely shrink between tasks, it inverts.

On MNIST, stripping the layer-specific sigmoid widths is catastrophic, accuracy collapses from 96.7% to 25.3%, near chance, while removing batch-centered error costs a rounding error and asymmetric initialization does nothing measurable. On CIFAR-10 the ranking flips: batch-centered error becomes the load-bearing component (−47.9 pp, collapsing four of five seeds), and sigmoid width slips to secondary.

Why? MNIST's well-separated features let the network learn even with a lopsided error signal, but without wide sigmoids the derivatives vanish. Post-hoc analysis shows a 25x gradient decay from output to first hidden layer. CIFAR-10's higher inter-class similarity makes the 9:1 one-vs-all error imbalance overwhelming, so centering the error becomes essential to stop the network from uniformly suppressing every class channel. Two tasks, two entirely different credit-assignment bottlenecks. Evaluate on either one alone and you would draw the opposite conclusion about which design choice is critical.

Reinforcement learning: where the tax gets cheaper

The same dual-stream core, dropped into Proximal Policy Optimization as ED-PPO, this time with ReLU activations and none of the classification tricks, tells a different story. On Google's Brax locomotion tasks it is competitive with standard backprop-based PPO, and on HalfCheetah it actually beats it (5,494 versus 3,520; p < 0.001) while matching DFA-PPO. This mirrors a broader trend in machine learning where the importance of different training components shifts drastically between domains, much like how agents trained in simulation flinch in the real world until noise injection is added to their training.

The open-ended Craftax benchmark is where the cross-domain pattern sharpens. ED-PPO trails backprop but beats DFA, whose random feedback pathways, perfectly adequate on supervised images, buckle on a complex exploration task. DFA is strong on classification and weak on hard RL. The cost of Dale's principle, in other words, is itself task-dependent.

The self-organizing catch

Two side effects hint at why biologically constrained networks behave the way they do. The 3:1 excitatory-to-inhibitory initialization drifts, during training, toward near-balance (~1:1) in hidden layers, with a depth-dependent inhibitory gradient, loosely echoing how cortical circuits mature toward balanced excitation and inhibition. And the non-negative floor prunes hard: 37.3% of weights hit the floor after training, with inhibitory feed-forward connections pruned most aggressively (up to 68.8%). Sakana frames that as a potential upside, Dale-compliant training may deliver model compression for free, since a weight pinned at zero cannot recover.

Why the constraint might earn its keep

None of this makes Dale-compliant learning competitive with backpropagation today. The paper is explicit: ED-PPO carries substantially higher variance, and the classification gap quantifies a real cost. The argument for pursuing it is downstream. Non-negative synaptic magnitudes with fixed-sign routing map naturally onto analog, photonic, and synapse-device neuromorphic hardware, where physical elements often encode only non-negative quantities and sign lives at the level of population identity. The excitatory/inhibitory split also offers an interpretability handle standard networks lack: when the model errs, you can ask whether a feature was falsely amplified or insufficiently suppressed. As other researchers have found, making AI systems better at evaluating their own decisions is often the bottleneck, not the learning rule itself.

The lasting takeaway is methodological, and it outlives Error Diffusion. A component that rescues one benchmark can be dead weight on another. Judging a biologically plausible learning rule, or arguably any architecture, on a single dataset does not just understate its limits; it can point researchers at exactly the wrong design lever. That is a cheaper lesson to learn from an ablation table than from a decade of misdirected work.