Distributional RL heads make risk claims that are mostly false. A new audit proves it.

Distributional reinforcement learning has become a dominant paradigm in modern AI. By replacing a single expected return with a full probability distribution over future rewards, these methods powered some of the most celebrated results in game-playing agents and now anchor a growing textbook theory. But a rigorous new study from researcher Hari Prasad asks a question few have measured directly: when a trained distributional agent reports that two actions differ in risk, is that report actually true?

The answer, across seven different algorithm, game combinations and thousands of audited states, is that it usually is not. The paper, delivered as a preprint and released with a full audit toolkit, provides what its author calls "a diagnosis that must precede any cure" for a flaw suspected for years but never quantified at the decision level.

The audit metric: isolating what matters

Prasad's central technical contribution is an audit metric called the excess Wasserstein gap, denoted Δ_x(s). For the two actions the agent ranks highest by mean at a given state, the metric measures the Wasserstein-1 distance between their return distributions minus the absolute difference in their means. When Δ_x(s) is zero, one action first-order stochastically dominates the other, every monotone risk functional ranks them identically, and a risk-sensitive policy would never deviate from a mean-greedy one. When Δ_x(s) is positive, the agent is claiming a genuine risk trade-off: neither action is safer at every quantile, and a user might rationally choose based on tail aversion.

"The metric isolates exactly the claims that carry decision content," Prasad writes. "It ignores errors both actions share, the audit is of relative risk claims, the kind acted upon."

This screening is paired with a statistical harness that includes permutation nulls, bootstrap refutation tests, and false discovery rate (FDR) control. Without that harness, the paper notes, the audit itself can manufacture false conclusions, two silent pitfalls of restart-based evaluation are documented and quantified.

Across three algorithms, the same signature

Prasad trained QR-DQN, C51, and IQN agents on three MinAtar games, Breakout, Seaquest, and Asterix, with multiple seeds per algorithm for a total of 33 runs. The results are strikingly consistent:

• 66% to 95% of the strongest claimed trade-offs (the top 2% of each agent's own Δ_x ranking) are refuted at 95% confidence.
• Essentially no claim is confirmable, zero out of 735 QR-DQN states survived FDR-corrected testing across all three games.
• The correlation between learned excess and ground-truth excess hovered near zero across all seeds and games (Pearson r ≈ 0.0).
• The placement of the agent's most confident claims is statistically indistinguishable from a truth-blind random shuffle.

"The heads place their strongest claims no better than truth-blind chance," Prasad concludes. The artifact is not a symptom of undertrained agents: it appears fully formed by 500,000 training steps, grows with the very claim strength a practitioner would rank on, is uncorrelated with final score, and reproduces on a pretrained near-state-of-the-art Atari agent.

Decision-level consequences: from beneficial to anti-predictive

To test whether the theoretical refutation translates to practical harm, Prasad compared three choosers of the safer action on the states the agents flagged as most risk-critical: the head's own CVaR ranking, a mean-greedy baseline that ignores risk, and ground truth. The outcome is environment-dependent in a way that leaves no reliable rule for practitioners.

In Breakout, the QR-DQN head was genuinely informative, choosing the safer action 81% of the time versus 44% for mean-greedy, with regret reduced from 1.36 to 0.20. In Seaquest, however, the head was anti-predictive: it chose the truly safer action only 34% of the time (below chance, p = 0.045), roughly tripling regret relative to ignoring risk entirely. In Asterix, it was a coin flip.

"A practitioner cannot tell which regime obtains," Prasad writes. "Advice that is sometimes helpful and sometimes inverted is harder to use than uniformly uninformative." C51 fared even worse, beaten by the mean-greedy baseline in all three games.

Repairs fail to restore usable risk claims

The paper tests three families of repairs suggested by the literature. Training the agent to be CVaR-greedy during learning (optimizing the tail) left the overclaiming unchanged. A five-member ensemble attenuated but did not calibrate the top claims. And recalibration via an isotonic map, which rescued the positive controls perfectly, passed the audit only by collapsing the top claims to noise, asserting zero real trade-offs.

"The head is not miscalibrated but uninformative," Prasad writes. "Missing information, not a fixable scale error." A distillation control confirmed the architecture can fit the true distributions when supervised directly with ground-truth targets, isolating the deficit to the bootstrapped training process rather than the network's capacity.

Two pitfalls that can fool researchers

The paper documents two methodological traps in restart-based evaluation that its author encountered while developing the audit. The first, RNG cloning, occurs when deep-copying an environment clones its internal random generator, causing every "random" rollout to replay an identical future. The failure is silent, directionally plausible, and survives code review easily. The second, the empirical Wasserstein noise floor, means that even two samples from the same distribution produce a positive measured excess that decays as n^−1/2. A naive fixed threshold can "confirm" two-thirds of claims that are pure noise.

Both pitfalls produced convincing but wrong audits before the final design. "The validation of distributional claims must be calibrated against a per-state null at the deployed sample size," Prasad warns.

What the finding means for the field

Distributional RL heads are increasingly used for three practical purposes: visualizing return distributions to interpret agent behavior, thresholding them to produce risk-sensitive policies via CVaR and other functionals, and monitoring them as safety signals in autonomous driving and safe-RL pipelines. All three uses share the assumption that where the learned distribution of one action differs in shape from another, the environment actually contains that risk structure.

"Face-value reads of distributional heads are least reliable at the states where they are most likely to be consulted," Prasad concludes, "and should be validated against ground truth before they are trusted."

For the growing body of work building uncertainty estimators, ensembles, and calibration layers around raw distributional heads, the audit provides both a measurement benchmark and a sobering baseline: an uninformative head gives post-hoc corrections no signal to work with. The paper releases its full audit toolkit and a concrete standard for future methods: an audit-passing head confirms its top claims.