Microsoft's experiential learning fix gives AI models a coach, not just a score

Reinforcement learning for language models has a bottleneck that anyone who has debugged a reward model knows well: the signal is too thin. A rubric evaluates a response, compresses that evaluation into a single scalar, and discards everything else about why one response was better than another. Two responses with the same score might have very different quality profiles, but the model never learns the difference.

A paper from Microsoft Research Asia, posted on arXiv this week, proposes a fix that treats that lost textual feedback as the primary signal rather than waste. Experiential Learning (EL) replaces the scalar reward with what the authors call experiential knowledge, structured, transferable textual feedback extracted by a coach model and then distilled into the policy model's weights during training.

The core insight is simple: if you already have a judge model evaluating responses against a rubric, that judge is sitting on rich information it never passes along. The authors repurpose it into a coach that outputs a short natural-language analysis of each response: what it did well, where it fell short, and what a stronger answer would look like. That text then conditions a teacher model that generates demonstration-quality responses. The policy model learns from those demonstrations through on-policy context distillation, essentially reading the coach's feedback and the teacher's improved response while updating its own parameters.

This is not the same pipeline as the On-Policy Skill Distillation framework we covered earlier this year, though both share the intuition that denser feedback matters. OPID extracts hindsight supervision from completed trajectories; EL extracts it from the judge's own evaluation text. The two could be complementary, as noted in the NousCoder-14B open RL pipeline that similarly aims to open the black box of reward design.

The paper tested EL across two policy families, the open-source Llama-3.1-8B-Instruct and a proprietary model, using feedback either from the policy itself or from GPT-4o as a coach. Across held-out and unseen open-ended tasks, EL consistently outperformed standard rubric-based RL.

The most interesting result is what happens outside the training distribution. Rubric-based RL tends to overfit to the evaluation criteria it trained on; given a slightly different rubric at test time, performance drops. EL's models held up better, suggesting that the textual feedback taught a more general understanding of quality rather than a mechanical mapping from rubric to score. The authors also report that EL mitigated reward hacking, the phenomenon where a model learns to exploit loopholes in the reward function rather than actually solving the task. For more on reward hacking persistence, see the two-word fix for reinforcement learning's self-sabotage problem.

This matters because reward hacking is a persistent problem in RL-based alignment. The previous fix addressed the issue from a different angle, preventing the optimization process from fighting itself. EL addresses it from the data side: if the reward signal carries more information, there is less room for the model to discover brittle shortcuts.

The tradeoff is compute. Running a coach model and a teacher model on every on-policy response adds to the training cost. The paper does not report wall-clock comparisons, but the pipeline clearly requires more forward passes per training step than a standard RL loop. The question is whether the improvement in generalization and robustness justifies the extra spend, and for any team training on non-verifiable tasks, the answer so far is yes.

Code is promised at Microsoft's repository under the project name EL. Once released, the real test will be whether the method scales to larger models and stays stable across different coaching models. For now, it offers a concrete path out of the scalar trap that has constrained post-training for open-ended tasks since RLHF became standard. As Qwen's application of RL to image generation shows, richer reward structures are becoming the trend across modalities.