Shared memory helps AI agents, but only until you hit five of them

A paper posted to arXiv on July 21, 2026 tackles a gap in reinforcement learning: what happens when actor-critic algorithms designed for single-agent parameterized action tasks get thrown into multi-agent settings without the usual centralized training scaffolding?

Clement Nyirenda extends three single-agent algorithms, Greedy Actor-Critic (GAC), Soft Actor-Critic (SAC), and Truncated Quantile Critics (TQC), into shared-experience multi-agent versions: MAGAC, MASAC, and MATQC. Instead of the centralized training, decentralized execution (CTDE) paradigm that dominates multi-agent RL, each agent keeps its own policy and value networks but dips into a single shared replay buffer.

The idea is pragmatic: let agents learn independently while pooling experience, cutting the coordination overhead CTDE often demands. The results, however, are uneven across algorithms. This pattern mirrors findings in other multi-agent systems, as seen with Sakana AI's Fugu orchestration system.

GAC jumps; SAC and TQC stall

On the Platform-v0 and Goal-v0 benchmarks, MAGAC consistently outperformed its single-agent counterpart. The gap was statistically significant across three-, five-, and ten-agent configurations, with one-way ANOVA and Tukey HSD post-hoc tests backing the comparison. For GAC, sharing a replay buffer was a clear win.

MASAC and MATQC showed more modest improvements. Their multi-agent versions did beat the single-agent baselines on average evaluation return, but the margins were narrow. The shared replay buffer gave SAC and TQC less room to exploit, possibly because both algorithms already include built-in mechanisms for exploration and stability that leave less low-hanging fruit for multi-agent architectures to pick.

The five-agent wall

The study tested each algorithm with three, five, and ten agents. The pattern was consistent: the jump from three to five agents brought measurable gains. The jump from five to ten did not. The extra agents barely moved the evaluation return while training time rose substantially, especially for MAGAC, which scaled the worst in computational cost.

That result matters for anyone designing real-world multi-agent systems with limited GPU budgets. The paper does not claim a universal threshold, but the data suggests that past five agents, shared-experience actor-critic architectures spend compute on redundancy rather than learning. The trade-off between performance and efficiency is sharp, a theme that also appears in broader AI scaling debates like the LiveBench diminishing returns analysis.

Nyirenda points out that the benchmarks used, Platform-v0 and Goal-v0, are discrete parameterized action environments. They require agents to pick both an action type and continuous parameters, making them closer to robotics control tasks than to board-game or Atari-style RL. The scalability ceiling could shift in different environments, but the paper stakes a clear empirical claim: for this family of tasks, five agents is a practical upper bound. This has direct implications for robotics pipelines, much like Xiaomi's open-source robotics model aims to address similar data bottlenecks.

Where shared experience works, and where it doesn't

The architectural choice is worth unpacking. The paper deliberately avoids CTDE, which requires a centralized critic that sees all agents' actions during training. Instead, each agent critic works from its own observation and action, with the replay buffer as the only shared resource. That keeps training simpler but also means agents cannot coordinate implicitly through a joint value function. The results suggest that for some algorithms (SAC, TQC) the loss of that coordination outweighs the benefit of pooled experience, while for GAC the pooling effect dominates.

A table summarizing the comparative results across configurations would clarify the pattern, but the paper's own figures, based on ten independent runs per configuration, show consistent variance that the statistical tests handle directly. The key takeaway is that algorithmic choice matters more than agent count once a minimum threshold is crossed. This finding echoes the message from analysis on parallel agent architectures, where concurrency alone doesn't guarantee better results.

For researchers building multi-agent RL systems in robotics, simulation, or continuous control, the practical implication is straightforward: start with three agents, test five, and do not assume ten will help. The extra compute is better spent tuning the single-agent backbone.