The 4B model that beats 32B ones by refusing to trust itself

Researchers have published a paper on arXiv describing RefineRL, a training method that teaches small language models to keep improving their own code solutions long after a first draft passes the visible tests. The key is a skeptical agent that refuses to trust its own outputs and keeps looking for bugs even when everything looks green.

The approach pairs a standard Qwen3-4B base model with two innovations. First, Skeptical-Agent, an iterative loop that runs the model's code against public test cases, then refuses to accept a passing grade as proof of correctness. Second, a reinforcement learning step that trains the model to actually use that loop, rather than ignoring it the way many finetuned models learn to game shorter reward paths.

After training, the compact 4B models equipped with Skeptical-Agent outperformed 32B models in competitive programming (CP) benchmarks. They also approached the single-attempt performance of 235B models, a category that includes frontier models from major labs. The paper reports gains on both the Qwen3-4B and Qwen3-4B-2507 checkpoints. See how another 33B model beat 137B models on coding without changing hardware.

This matters because the CP domain has become a standard stress test for reasoning in LLMs. Single-attempt evaluation dominates the literature. Most papers test whether a model can solve a problem in one shot. RefineRL asks a different question: can a model learn to use the time it already has?

The skepticism mechanic is where the method departs from prior work. Standard RL for coding problems rewards a solution that passes the tests and stops. That teaches models to produce first drafts that barely clear the bar. Skeptical-Agent explicitly does not stop after a pass. It actively searches for counterexamples, edge cases, or assumptions that could make the solution break under a different input. The paper describes this as maintaining a skeptical attitude toward its own outputs. This echoes the concept of systematic distrust explored in a broader framework from DeepMind: why AI is brilliant at some things and terrible at others.

The RL component uses standard RLVR data, meaning pairs of problems and verifiable answers without requiring handcrafted reward signals or human annotations along the way. That keeps the training pipeline light. The researchers argue this is a practical advantage: any lab with a programming problems dataset can reproduce the setup. Recent open-source efforts like NousCoder-14B show similar RL pipelines reaching high Codeforces solve rates, though targeting a larger parameter count.

Whether the approach generalizes beyond competitive programming remains an open question. CP provides clean feedback: a solution either passes the hidden tests or it doesn't. That binary signal is exactly what skeptical self-refinement needs. For tasks with fuzzy or subjective correctness criteria, the same technique may not translate. But the paper suggests that the principle, iterative refinement driven by systematic distrust of one's own output, may be more broadly applicable than just coding.

The work also adds to a growing body of evidence that model size alone is not the only path to better reasoning. On-policy skill distillation, outcome-based RL, and now skeptical refinement all show that training methodology can sometimes close the gap between a 4B and a 32B model more effectively than scaling parameters. The scaling laws are real, but they may have a neglected variable: how thoroughly the model reexamines its own first attempt. Several studies confirm this trend, see why an agent with a learning loop changes the autonomous AI game and how four agents beat one by talking to themselves.

RefineRL is available on arXiv under the identifier 2604.00790.