The two-word fix for reinforcement learning's self-sabotage problem

Reinforcement learning from human feedback (RLHF) has become the standard way to align large language models with human preferences. But as models train on an ever-widening set of objectives, instruction following, writing style, factual accuracy, code correctness, the reward signals that drive learning get messy and often contradictory. A new paper from an international research team proposes a fix that directly addresses these instabilities.

The multi-reward problem

In multi-task reinforcement learning, each prompt produces its own reward vector, combining binary judgments (did the model follow the instruction?), fractional scores (how grammatical is the output?), and continuous metrics (embedding similarity). These rewards often operate at vastly different scales and distributions. Worse, reward dimensions can be correlated: a model that writes well may also score higher on instruction following simply because coherent language tracks with completing the task. Standard advantage estimation methods, which compute a single scalar from weighted sums, amplify noise from both heteroskedasticity and redundancy. the verification horizon

"The central difficulty is that a naive scalarization of rewards before advantage computation produces unstable gradient updates," the authors write. "This is especially acute when the reward distribution changes across prompts and training stages."

The researchers identify two failure modes: magnitude disparity (a fractional reward ranging 0-1 paired with a continuous reward ranging 0-100) and correlation redundancy (two reward dimensions that encode overlapping information, leading to overconfident gradient steps). on-policy skill distillation

RDPO's two-stage architecture

Reward-Decorrelated Policy Optimization tackles both problems in sequence. The first stage applies Magnitude-Aware Quantile normalization (MAQ), which sorts all rewards for a given dimension across the batch, anchors them at their median, and scales them by the interquartile range. Unlike standard quantile normalization, MAQ preserves magnitude information by tracking the median as a learned parameter, which lets the model distinguish between a consistently high reward and one that is sporadically high.

The second stage applies Mahalanobis whitening within each active reward subspace. The researchers group reward dimensions that are active for a given prompt, a task-appropriate subset of the full reward vector, and compute the Mahalanobis distance, which effectively decorrelates the dimensions before aggregation into a scalar advantage. This prevents redundant signals from being double-counted in the gradient update.

"The Mahalanobis step is crucial because it acts as a per-batch correction," the authors explain. "Reward correlations are not static, they shift as the model's policy changes. Online decorrelation captures this dynamic."

Results on LongCat-Flash

The researchers integrated RDPO into the post-training pipeline of LongCat-Flash, an existing long-context model. Compared to standard advantage estimation, RDPO improved instruction following by 8.3 percentage points on the MT-Bench instruction-following subset, raised writing quality scores by 6.1 points on a proprietary writing evaluation suite, and reduced sensitivity to hard prompt variants by 12% in adversarial testing. On standard reasoning benchmarks (GSM8K, MATH) and coding evaluations (HumanEval, MBPP), performance remained within statistical noise of the baseline, no degradation from the decorrelation step. IFBench instruction following

An ablation study showed that MAQ and Mahalanobis whitening are individually beneficial: MAQ alone recovered 60% of the total gain, and the whitening step added the remainder. Removing both dropped performance to the original baseline.

Broader implications

The paper arrives at a moment when RLHF pipelines face increasing scrutiny, with debates over reward hacking, reward model collapse, and the fragility of multi-objective optimization. RDPO offers a relatively light-weight plug-in: it does not require retraining the reward model, only modifying the advantage computation, which makes it feasible for existing RLHF pipelines. the 95% gap

"The beauty of the approach is that it treats reward processing as a first-class optimization problem rather than just a pre-processing step," says Dr. Sarah Chen, a reinforcement learning researcher not affiliated with the study. "Many labs simply normalize rewards to mean zero, unit variance, and hope for the best. RDPO explicitly designs for the two hardest cases."

What remains unsolved

The paper does not address reward model accuracy itself, RDPO only improves the consumption of rewards, not the quality of the reward signal. If the reward model is biased or misaligned, decorrelated or not, the policy will still optimize for the wrong target. There is also the computational overhead of computing per-batch covariance matrices, which grows quadratically with the number of active reward dimensions and could become a bottleneck in very high-dimensional spaces.

The authors acknowledge that their method assumes reward dimensions can be reasonably partitioned into active subspaces, a design choice that may not be straightforward for tasks with deeply entangled objectives. MiniMax M3

"We see RDPO as a foundation for more principled multi-reward RL, not as a final solution," they conclude. "Understanding when rewards should be decorrelated and when they should be kept coupled is an open research direction."