The 4B model that beats 32B ones by refusing to trust itself

RefineRL, a new training method described in an arXiv paper, teaches small language models to keep improving their own code solutions even after a first draft passes the visible tests. The key is a skeptical agent that actively hunts for bugs after tests turn green, refusing to trust its own work. This aligns with validation-focused agent architectures explored in recent safety research on separate validation brokers.

The approach pairs a standard Qwen3-4B base model with two innovations. First, Skeptical-Agent, an iterative loop that runs the model's code against public test cases, then refuses to accept a passing grade as proof of correctness. Second, a reinforcement learning step that trains the model to actually use that loop, unlike many finetuned models that learn to game shorter reward paths.

After training, the compact 4B models equipped with Skeptical-Agent outperformed 32B models in competitive programming (CP) benchmarks. They also approached the single-attempt performance of 235B models, a category that includes frontier models from major labs. The paper reports gains on both the Qwen3-4B and Qwen3-4B-2507 checkpoints. This adds to the debate about coding benchmarks themselves, as discussed in the limits of HumanEval as a measure of real coding ability.

Competitive programming has become the standard stress test for reasoning in LLMs, though benchmark integrity remains an open concern as audits continue to find contamination in agent benchmarks. Single-attempt evaluation dominates the literature. Most papers test whether a model can solve a problem in one shot. RefineRL asks a different question: can a model learn to use the time it already has?

The skepticism mechanic is where the method departs from prior work. Standard RL for coding problems rewards a solution that passes the tests and stops. That teaches models to produce first drafts that barely clear the bar. Skeptical-Agent explicitly does not stop after a pass. It actively searches for counterexamples, edge cases, or assumptions that could make the solution break under a different input. The paper frames this as a form of skepticism directed at the model's own outputs. This approach mirrors a shift from simple reward signals to richer training feedback, as seen in Microsoft's coach-based experiential learning framework.

The RL component uses standard RLVR data, meaning pairs of problems and verifiable answers without requiring handcrafted reward signals or human annotations along the way. That keeps the training pipeline light. The researchers argue this is a practical advantage: any lab with a programming problems dataset can reproduce the setup. Recent open-source efforts like NousCoder-14B show similar RL pipelines reaching high Codeforces solve rates, though aiming for a larger parameter count.

Whether the approach generalizes beyond competitive programming remains an open question. CP provides clean feedback: a solution either passes the hidden tests or it doesn't. That binary signal is what skeptical self-refinement needs. For tasks with fuzzy or subjective correctness criteria, the same technique may not translate. But the paper argues that the principle iterative refinement driven by systematic distrust of one's own output may be more broadly applicable than just coding.

The work adds evidence that model size alone is not the only path to better reasoning. On-policy skill distillation, outcome-based RL, and now skeptical refinement all show that training methodology can close the gap between a 4B and a 32B model more effectively than scaling parameters. The scaling laws are real, but they may have a neglected variable: how thoroughly the model reexamines its own first attempt. This pattern is visible across recent research, from Nvidia's 4B edge model that punches above its weight to the broader shift toward orchestration over raw parameter count.

RefineRL is available on arXiv under the identifier 2604.00790.