Qwen just taught diffusion models a trick from the LLM playbook: RL beats supervised fine-tuning

Reinforcement learning from human feedback, a technique best known for aligning large language models, is now being applied directly to diffusion models. The Qwen team's latest technical report, published on arXiv on June 25, describes Qwen-Image-2.0-RL, a post-training pipeline that uses RLHF and on-policy distillation to improve both visual quality and instruction following in the existing Qwen-Image-2.0 model. The approach builds on earlier work in the space, including Nous Research's fully open RL pipeline for coding, though applied here to image generation rather than competitive programming.

The central challenge the team tackled is that post-training for generative models typically relies on supervised fine-tuning, which can lead to "reward hacking", the model learns to exploit the training signal rather than genuinely improving. RL offers a way around this by optimizing against a reward function directly, but building a reliable reward system for image generation is harder than it is for text, since image quality is subjective and multi-dimensional. As one recent analysis found, the reward signals that drive alignment often work against each other, requiring careful calibration to avoid reinforcement learning's self-sabotage problem.

Composite reward models

To provide the needed signals, the researchers constructed task-specific composite reward models, fine-tuning existing vision-language models with a pointwise scoring paradigm and chain-of-thought reasoning. For text-to-image generation, the reward system covers three dimensions: prompt alignment (how well the image matches the text), aesthetic quality (subjective appeal and composition), and portrait fidelity (how well faces are rendered). For image editing tasks, the reward models evaluate instruction-following accuracy and face identity preservation, two aspects where diffusion models often struggle, particularly when asked to modify specific features while keeping a person recognizable.

Building this reward scaffolding required adapting the vision-language models to output a numeric score per image, along with a chain-of-thought reasoning step that makes the scoring partially transparent. The report notes that this scoring paradigm outperformed pairwise preferences on consistency and reliability.

Scalable GRPO training

With the reward system in place, the team developed a scalable training framework based on Group Relative Policy Optimization (GRPO), a variant of reinforcement learning that compares groups of completions rather than individual ones. The pipeline incorporates three engineering choices aimed at preserving pre-trained knowledge while learning new behaviors:

• Hybrid classifier-free guidance (CFG): The model switches between RL-trained and frozen CFG scales during training to avoid forgetting the original distribution, a known issue when applying RL to diffusion models from scratch.
• Intra-group reward range filtering: Prompt-level generation quality varies, and a batch can contain both excellent and poor outputs for the same prompt. The team filters prompts by the range of rewards within their group, discarding those where all variants score similarly or where the spread is too narrow to provide a useful gradient.
• Per-category reward weight calibration: Not all reward dimensions matter equally for every prompt. The system dynamically tunes the weight assigned to alignment vs. aesthetics vs. portrait fidelity based on the prompt category, preventing any single metric from dominating the training signal.

The report does not disclose model size or training compute, but the approach is designed to be applied on top of an existing pre-trained and supervised fine-tuned diffusion model, meaning the RL stage adds marginal cost relative to training from scratch.

On-policy distillation merges two policies

A notable design decision is the separation of the training stage. The team first trained two separate RL policies, one specialized for text-to-image generation, one for image editing, and then merged them into a single model using on-policy distillation. This last stage treats the two specialized models as teachers and a fresh copy of the base model as the student, training it to match the teachers' trajectory-level velocity, essentially, the direction and magnitude of updates at each step, rather than just final outputs. This avoids the catastrophic forgetting that often happens when fine-tuning a single model sequentially on two different tasks.

The resulting student model, Qwen-Image-2.0-RL, was evaluated on Qwen-Image-Bench, the team's own benchmark suite, where it scored 57.84 overall, a +2.61 improvement over the base Qwen-Image-2.0 model. In the text-to-image arena, it achieved an Elo rating of 1193 (+78), and in the image editing arena, 1349 (+93).

The Elo gain in the editing arena is the larger of the two, perhaps reflecting the fact that guided editing, modifying a specific region of an existing image without degrading the rest, remains one of the harder problems for diffusion models and thus offers more room for improvement through RL-based post-training. The approach echoes the multi-agent strategies seen in other recent work, where specialized agents or policies are trained separately and then merged, rather than attempting to optimize for everything at once.

Open questions

The report is a technical paper rather than a product launch, and several questions remain unanswered. The composite reward models were fine-tuned on an undisclosed dataset of human preferences, and the evaluation metrics are all internal to the Qwen team. Independent verification, particularly on the editing benchmark, where face identity preservation under editing is notoriously hard to evaluate automatically, would strengthen the claims.

There is also no comparison to alternative post-training methods like Direct Preference Optimization (DPO) applied to diffusion models. The report briefly mentions having tried DPO-style losses but does not provide quantitative comparisons. Given DPO's growing popularity in the vision space, a head-to-head would help the community determine when the additional complexity of GRPO is worth it. The same team's earlier work on treating environments as language problems suggests a broader strategy of adapting RL techniques across domains.

The technique faces domain-specific limitations in the report's own framing. The reward models for portrait fidelity assume faces are a primary subject; for scenes where no human faces appear, that dimension is simply zero-weighted. More broadly, constructing robust reward models for arbitrary prompts, especially those involving abstract concepts or unusual aesthetics, remains unsolved, and the system's performance likely degrades on prompts far from the fine-tuning distribution. This mirrors the broader challenge of what VLMs miss about human creativity, where synthetic evaluation methods fail to capture certain dimensions of human aesthetic judgment.