A panel of AI agents can agree and still be wrong, paper argues

Ask several large language models the same question and they will not always reply with the same answer. What happens next decides whether that variety helps or hurts. A vote, an electoral rule, or an LLM judge settles the split, and the reasoning stops there. A paper submitted to arXiv on 10 September 2026 argues the trouble starts at exactly that step.

The paper is titled When Agents Disagree: Bayesian Backward Reasoning as a Label-Free Anchor for Multi-Agent Collective Decision-Making. Its target is not the individual model but the layer that sits on top of a pool of them.

Why a vote inherits the same error

Voting, electoral rules, and LLM judges look like different mechanisms for resolving a disagreement. The authors group them anyway. All three reason forward: they take evidence and map it to a label in one direction. The estimates that come out of that process share what the paper calls the same evidence-to-label factorization, and approximations built on a single factorization can carry errors that are correlated instead of independent. Pooling models to beat any one of them is common practice, though the usual yardstick is cost, as CoBa's comparison against best-of-16 majority voting illustrates.

The distinction carries the whole argument. If five agents fail for unrelated reasons, combining them cancels noise. If they fail for the same reason, the combined answer agrees with the error and looks confident while doing it. The paper does not claim forward methods always fail together, only that they can, and that the failure mode is invisible from the outputs alone. The same blind spot shows up in single agents: research on automated hypothesis testing found agents can execute an analysis flawlessly and still reach the wrong conclusion, because the benchmarks score the execution rather than the judgment.

Running the model backwards

The proposed alternative is to build a second estimate from the other direction. For each instance, the authors construct a reverse posterior through Bayesian backward reasoning from an explicit likelihood. In their framing, the forward and reverse posteriors are two differently factorized approximations of the same underlying posterior, which is what makes them useful as a pair.

The reasoning then turns probabilistic. Estimates that arrive through different factorizations "may tend to share the same error less often," the abstract states, which is a hypothesis about error independence rather than a demonstration of it. To put it to work, the authors use Jensen-Shannon divergence to rank agents by cross-path consistency, a measure of how closely an agent's forward answer lines up with the reverse path.

Three ways to spend a consistency signal

That ranking feeds three aggregation strategies, all evaluated on DDXPlus across five LLM backbones. Picking the right quantity to optimize is where routing designs tend to go wrong, as work on per-call routing has argued: scoring each step in isolation can miss the task-level outcome altogether.

Strategy	Mechanism	Reported result
MinJS	Hard selection: pick agents by consistency score	Outperforms random selection across all five backbones
FwdJS	Soft reweighting: keep every agent and adjust its influence	Generally improves on the strongest baseline
LogLin	Log-linear fusion of the ranked pool	Best performance among the evaluated methods

The spread of those gains matters more than the headline. LogLin's largest improvements land on the subset of cases where the agents disagree. A method that only helped where agents already agreed would undercut its own premise, so this is the result the paper leans on. Reported aggregate wins in multi-agent setups tend to bury the same distribution question, a reason Qoder's multi-agent experiment drew scrutiny over its headline error-reduction figure.

Where the reverse anchor is weak

One reported detail cuts against a simple reading of the method. Standing alone, the reverse posterior is less accurate than the forward-only alternatives it is meant to improve on. The authors do not hide this. They argue its value lies in being a more useful anchor than a forward-only alternative, because it contributes information the forward pool cannot produce on its own.

What the DDXPlus result does not say

The abstract contains no numbers. There is no accuracy figure, no margin over a baseline, no confidence interval, and no per-backbone result, and the five LLM backbones are never named. "Best performance among the evaluated methods" is the strongest quantitative claim in the text, and it is a ranking rather than a measurement. Anyone weighing the approach against a production pipeline should treat that as a gap to check in the full paper, not as a settled number. The caution generalizes: across agent tooling, shared trust assumptions between vendors can leave a downstream agent relying on a compromised tool with no signal in the output it gets back.

There is a second thread worth pulling. The method is presented as a label-free anchor, meaning it ranks agents without ground-truth answers. But the paper adds that when labeled data are available, a lightweight two-stage calibration can refine the reverse anchor further and improve aggregation. So the technique works without labels and works better with them, which leaves the choice of variant tied to data a team may or may not have.

The gap the paper points at is structural rather than exotic. Aggregation methods combine estimates without separating independent failures from shared ones, and a confident majority can hide the difference. Bayesian backward reasoning is one way to ask whether a pool is diverse in the way that matters. Whether that question holds up beyond DDXPlus is left to the full text and to anyone who tries to reproduce it.