RL's sparse-reward blind spot meets SEED: agents write their own lessons

Reinforcement learning for large language models has a built-in blind spot. An agent that takes twenty actions gets exactly one piece of feedback: whether the whole episode eventually worked. That tells it little about which of the twenty decisions mattered. Outcome-based RL works well enough to be standard, but it leaves a gap between the episode-level reward and the token-level policy that produced it. Long-horizon tasks are where the gap bites hardest, the same failure mode that tripped up autonomous agents in the Picbreeder reproduction experiment.

A team from the University of Hong Kong and Shanghai AI Laboratory proposes a workaround that sounds obvious after the fact: let the model read its own completed trajectories and write down what it learned, then feed those lessons back during training. They call it SEED (Self-Evolving On-Policy Distillation), and the preprint posted to arXiv on July 16, 2026, reports consistent gains on both text-based and vision-based agentic tasks.

From sparse reward to dense signal

The core idea is that a model, when told to examine a trajectory it just finished, can produce natural-language skills that describe reusable workflows, decisive observations, or rules for avoiding a repeated failure. The skills are not static templates written by a human. They emerge from whatever the policy currently does. A model that keeps overshooting in navigation might write a skill like "after three failed moves, check whether the target is still in the visible area." It is a lightweight version of the move toward organizing agents around reusable capabilities, a pattern formalized in skill-based agent architectures.

SEED turns those skills into a dense token-level signal. It re-scores the same actions under two contexts, one plain and one augmented with hindsight skills, and uses the difference in token probabilities as a distillation target. That target pushes the policy toward the behavior its own analysis describes, while the original outcome-based RL objective stays in the loop. Sparse reward plus dense hint, rather than either one alone. The same blend of on-policy distillation and reward-based training sits at the center of Qwen-Image 2.0's RL pipeline.

Self-evolving supervision

The self-evolving part is what keeps the scheme honest. The policy collecting trajectories today is better than the one that collected them yesterday, so SEED does not freeze the analysis module after the first pass. The same model acts as both actor and analyst of its own behavior; as the policy improves, the skills it extracts improve with it. The distillation signal stays close to the current trajectory distribution instead of reflecting an outdated snapshot.

That avoids a recurring failure in hindsight relabeling and skill extraction, where auxiliary supervision drifts out of sync with what the model is actually producing. SEED's feedback loop holds the two together. The philosophy is close to the one behind a paper on evolving distillation contexts, which argues that frozen context stops adding signal once it is distilled into a student.

What the experiments show

The paper reports results in text-based environments (ALFWorld, ScienceWorld) and vision-based ones (Craftax, a dungeon-crawler benchmark). SEED improves over pure outcome-based RL and over baselines using fixed, pre-extracted skills in every environment. The gains are largest on long-horizon tasks where the supervision gap is widest: ScienceWorld tasks requiring fifteen-plus steps show a 22% improvement in success rate over the best RL-only baseline.

Generalization to unseen scenarios also improves, measured by holding out environment configurations during training. The authors argue the hindsight skills capture genuinely reusable patterns rather than task-specific memorization, and the distillation step transfers those patterns to the policy without overfitting to the training distribution. That is the same bet behind Microsoft's LLM-as-a-Coach, which pulls transferable lessons out of individual episodes instead of just assigning a score.

One table tells the story

Across four environments and multiple difficulty levels, SEED beats the next-best method in nine of twelve reported comparisons. The losses are confined to short episodes, where the supervision gap is minimal anyway.

The improvements are not large enough to call a breakthrough. They are consistent, and they compound as episodes get longer. That is the kind of signal practitioners care about: better results without a new architecture or a bigger training budget. Similar cost-efficient gains were the story behind Mistral's OCR 4 benchmark audit.

Open questions

The paper does not answer everything. Nothing external validates the skills the hindsight module produces; the model can write a plausible-sounding lesson that is actually wrong. The authors note that a bad skill on average contributes no useful gradient, since the distillation signal is computed from probability shifts, but they do not measure how often the model produces misleading skills or whether that failure mode is common enough to matter at scale.

The computational cost of running the analysis module on every trajectory is also nontrivial. The authors call the overhead manageable because the analysis is a single forward pass per trajectory, but on very long episodes in complex environments, that extra pass adds up.

Still, SEED is one of the more practical proposals in the recent wave of agentic RL methods. It needs no human annotation and it slots into standard RL pipelines without invasive changes. It follows the example of OpenForgeRL's no-touch training setup, which also trains agents without reworking how they run. For teams whose agents fall apart on long-horizon tasks, it is worth a conversation.