RDPO: The two-word fix for reinforcement learning's self-sabotage problem

RLHF is the standard method for aligning LLMs with human preferences. But the more objectives you pile on, instruction following, writing style, factual accuracy, code correctness, the more the reward signals cross wires and contradict each other. A new paper from a multinational team proposes a fix that tackles these instabilities head-on.

The multi-reward problem

In multi-task reinforcement learning, each prompt generates its own reward vector: a mix of binary flags (did the model follow the instruction?), fractional scores (how grammatical is the output?), and continuous metrics (embedding similarity). These rewards often live on wildly different scales and distributions. Worse, reward dimensions can correlate, a model that writes well may also score higher on instruction following simply because coherent prose tracks with completing the task. Standard advantage estimation methods, which collapse everything into a single weighted scalar, amplify noise from both heteroskedasticity and redundancy, a problem explored in detail by the verification horizon.

“The central difficulty is that a naive scalarization of rewards before advantage computation produces unstable gradient updates,” the authors write. “This is especially acute when the reward distribution changes across prompts and training stages.”

The researchers pin down two failure modes: magnitude disparity (a fractional reward ranging 0, 1 paired with a continuous reward ranging 0, 100) and correlation redundancy (two reward dimensions encoding overlapping information, leading to overconfident gradient steps). Similar challenges have been addressed using on-policy skill distillation.

RDPO's two-stage architecture

Reward-Decorrelated Policy Optimization tackles both problems in sequence. First, Magnitude-Aware Quantile normalization (MAQ) sorts all rewards for a given dimension across the batch, anchors them at their median, and scales by the interquartile range. Unlike standard quantile normalization, MAQ preserves magnitude information by tracking the median as a learned parameter, so it can distinguish between a consistently high reward and one that spikes only occasionally.

Second, Mahalanobis whitening operates within each active reward subspace. The team groups reward dimensions that are relevant for a given prompt (a task-specific subset of the full vector) and decorrelates them via the Mahalanobis distance before aggregation into a scalar advantage. This prevents redundant signals from being double-counted in the gradient update.

“The Mahalanobis step is crucial because it acts as a per-batch correction,” the authors explain. “Reward correlations are not static; they shift as the model's policy changes. Online decorrelation captures this dynamic.”

Results on LongCat-Flash

When the team integrated RDPO into LongCat-Flash’s post-training pipeline, instruction following improved by 8.3 percentage points on the MT-Bench instruction-following subset, writing quality scores rose 6.1 points on a proprietary evaluation suite, and sensitivity to hard prompt variants dropped 12% in adversarial testing. On standard reasoning benchmarks (GSM8K, MATH) and coding evaluations (HumanEval, MBPP), performance stayed within statistical noise of the baseline, no degradation from the decorrelation step. These improvements align with findings from IFBench.

An ablation study showed that MAQ alone recovered 60% of the total gain; the whitening step added the remainder. Removing both dropped performance to the original baseline.

Broader implications

The paper arrives as RLHF pipelines enter a period of heavy scrutiny, debates over reward hacking, reward model collapse, and the fragility of multi-objective optimization are ongoing. RDPO offers a relatively light-weight plug-in: it requires no retraining of the reward model, only a modification to the advantage computation, making it feasible for existing RLHF pipelines. The persistence of these foundational issues is detailed in the 95% gap.

“The beauty of the approach is that it treats reward processing as a first-class optimization problem rather than just a pre-processing step,” says Dr. Sarah Chen, a reinforcement learning researcher not affiliated with the study. “Many labs simply normalize rewards to mean zero, unit variance, and hope for the best. RDPO explicitly designs for the two hardest cases.”

What remains unsolved

RDPO does not address reward model accuracy itself, it only improves the consumption of rewards, not the quality of the signal. If the reward model is biased or misaligned, decorrelated or not, the policy will still optimize for the wrong target. There is also the computational overhead of computing per-batch covariance matrices, which grows quadratically with the number of active reward dimensions and could become a bottleneck in very high-dimensional spaces.

The authors acknowledge that their method assumes reward dimensions can be reasonably partitioned into active subspaces, a design choice that may not be straightforward for tasks with deeply entangled objectives, a challenge also noted in the MiniMax M3 analysis.

“We see RDPO as a foundation for more principled multi-reward RL, not as a final solution,” they conclude. “Understanding when rewards should be decorrelated and when they should be kept coupled is an open research direction.”