AI benchmarks
23 published articles
Autonomous coding agents · arXiv preprint
Harness-of-Harness claims a 52.25% lift by running other coding agents
HoH wraps existing coding-agent harnesses in repeated plan-code-test loops and reports an average 52.25% relative gain across three benchmark suites. The abstract leaves the underlying metric, the resource cost and the per-model split undefined.
2026-09-25
OpenBMB's MiniCPM5-2B: 2.52B parameters, 4B ambitions
A 2.52B model outscores 4B rivals, on a benchmark set its own maker ran
A 2.52B Apache-2.0 model that beats larger open baselines on coding, tool use and long context, with a benchmark table OpenBMB says mixes internal and third-party runs.
2026-09-22
Machine-native AI and the benchmarks behind it
TypeSafe AI says it's 193x faster than LLMs. Its own proof shows 75x.
TypeSafe AI calls Jev the first System One Model: a system built to return typed decisions to software, trained with an algorithm it names RLCD. The launch page claims zero hallucinations and a 238x price advantage over a single named baseline, Claude Fable 5.1.
2026-09-21
AI research: industrial content risk control
SIRF's 15.1-point gain lives in the weights, not the prompt
SIRF moves a platform's moderation rules out of the prompt and into the model's weights through continued pretraining on synthetic policy data. The paper's real selling point is its control: one variable changed, 15.1 points gained.
2026-09-19
Open Source AI: Alibaba Opens the Max Tier
Qwen 3.8-Max: Alibaba's most powerful model is now free to download
Alibaba is open-sourcing Qwen 3.8-Max, its most capable model ever: a 2.4T-parameter MoE that beats GPT-5.6 Sol on SWE-bench Pro, PaperBench, and IFBench. We break down the benchmark caveats and what a 95B-active open flagship means for developers.
2026-08-16
Open-source AI
Qwen3.8-Max beat 458 human teams in 24 hours, working alone
Qwen3.8-Max beat 458 of 526 human teams in a 24-hour contest while working alone, and Alibaba will open-source its weights next week. Every number is self-reported so far, which is exactly why the autonomy claims deserve scrutiny.
2026-08-03
Special Report: AI Evaluation
The State of AI Benchmarking in 2026: Inside the Collapse of Static Tests and the Systems Built to Replace Them
A comprehensive tour of the 2026 AI benchmarking landscape: why MMLU, GSM8K, and HumanEval broke, how dynamic benchmarks and expert exams like HLE and GPQA Diamond replaced them, what agentic and jagged-intelligence testing reveals, and how human preference, LLM judges, and production observability now round out the full evaluation stack.
2026-08-03
Benchmark Analysis
LiveBench Refuses to Sit Still. That's the Whole Point
LiveBench replaces a fixed answer key with a continuously refreshed pool of coding problems, repositories, and prediction questions, sidestepping the contamination that undermined MMLU and GSM8K. It's part of a broader shift toward dynamic evaluation alongside LiveCodeBench, ForecastBench, and LLMEval-Fair.
2026-08-03
Benchmark Analysis
5 Million Votes, 25 Elo Points: Inside the Chatbot Arena Leaderboard Nobody Can Shake
LMSYS Chatbot Arena ranks models by blind human preference on an Elo scale, with nearly five million votes now packing the top labs into a 25-point band. The LLM-as-a-Judge methods that scale this kind of evaluation carry their own documented biases toward verbosity and position.
2026-08-02
Benchmark Analysis
On SWE-bench Verified, Top Models Hit 96%. On Private Enterprise Code, They Barely Clear 23%
SWE-bench Verified makes frontier models look close to solving real-world software engineering, with top scores above 95%. SWE-bench Pro, run on private enterprise repositories, drops those same models to 23% or lower, exposing how much of the Verified score depended on public data exposure.
2026-08-02
Benchmark Analysis
GPQA Diamond Was Built to Be Google-Proof. Frontier Models Are Now Clearing 95% Anyway
GPQA Diamond was designed so search-equipped humans can't reliably answer its expert-level science questions. Frontier models are now clearing 89% to 96%, pushing the benchmark toward the same saturation that retired MMLU.
2026-08-01
Benchmark Analysis
Humanity's Last Exam Opened With a 2.7% Score. The Best Models Still Can't Break 65%
Humanity's Last Exam was built by CAIS and Scale AI to succeed MMLU as the hardest general LLM benchmark. Two years on, even Claude Opus 5's leading 64.7% score sits well below the 90% human expert baseline.
2026-07-31