SWE-bench
3 published articles
Benchmark Analysis
On SWE-bench Verified, Top Models Hit 96%. On Private Enterprise Code, They Barely Clear 23%
SWE-bench Verified makes frontier models look close to solving real-world software engineering, with top scores above 95%. SWE-bench Pro, run on private enterprise repositories, drops those same models to 23% or lower, exposing how much of the Verified score depended on public data exposure.
2026-08-02
Benchmark Analysis
HumanEval Measured Whether AI Could Code. It Never Asked Whether the Code Was Real Work
HumanEval's 164 function-completion problems became the standard test for AI coding ability, but memorization and its narrow scope left a wide gap between passing the benchmark and handling a real codebase, a gap SWE-bench was built to expose.
2026-07-30
AI Models
The 33B model that just beat 137B models on coding benchmarks without changing hardware
Poolside's Laguna XS 2.1 improves SWE-bench Multilingual by 5.4 points to 63.1% while keeping the same 33B-total-3B-activated MoE architecture. The release includes quantized checkpoints, speculative decoding draft models, and an OpenMDW-1.1 license, making local AI coding more practical.
2026-07-21