Benchmarks & Tests
Comparisons, performance tests and model evaluations.
23 published articles
Benchmark Analysis
HumanEval Measured Whether AI Could Code. It Never Asked Whether the Code Was Real Work
HumanEval's 164 function-completion problems became the standard test for AI coding ability, but memorization and its narrow scope left a wide gap between passing the benchmark and handling a real codebase, a gap SWE-bench was built to expose.
2026-07-30
AI Evaluation
Two out of three AI agents are cheating on benchmarks, a new audit finds
HackDetect audits 15 agent benchmarks and finds 67% of Frontier Science runs are contaminated. Score inflation ranges from 0.45 to 1.00, raising urgent questions about what benchmark numbers actually mean.
2026-07-30
Grade-School Math Benchmark Quality Audit
GSM8K Was Supposed to Test Grade-School Math. Up to 42% of Its Questions Don't
GSM8K became a standard test of arithmetic reasoning in LLMs, but audits have found error rates in its question set as high as 42%, undermining what its scores actually measure.
2026-07-29
Benchmark Analysis
MMLU Ruled AI Benchmarking for Years. Then Models Started Acing the Answer Key
MMLU defined a generation of AI benchmarking, but training-data contamination and a flawed answer key have pushed frontier labs toward dynamic successors like HLE and GPQA Diamond.
2026-07-28
Benchmark
LLMs can describe data. They cannot reason through it. A new benchmark proves the gap is real.
SDABench, a new capability-oriented benchmark spanning six core scientific reasoning skills and five domains, tests 15 LLMs and finds that models are strong on descriptive analysis but collapse on inferential and causal tasks. The paper provides a five-stage error analysis framework to localize failures.
2026-07-25
Benchmark & Tests
Your graph model breaks on dirty data. A new benchmark shows exactly where.
OpenRTAG, a benchmark from an academic team, organizes TAG quality issues into a 3×3 taxonomy covering text, structure, and label degradation. It tests traditional GNNs, large language model-enhanced GNNs, and graph foundation models across nine datasets, revealing different sensitivity patterns that prior fragmented studies missed.
2026-07-24
Agent evaluation
Your AI agent keeps failing? It might be the harness, not the brain
PawBench, an open-source benchmark from the AgentScope team, systematically evaluates models and agent harnesses together. Results show that harness design can swing scores by over 11 points for smaller models, exposing a blind spot in how AI agents are currently judged.
2026-07-22
Agentic coding
Grok 4.5 just broke the coding agent leaderboard: the lead is real, the margins are tiny
Grok 4.5 now leads the SWE Marathon leaderboard, beating Claude 4 Opus and GPT-5. The benchmark tests real software engineering skills: bug fixes, feature additions, and code understanding across real repositories. The margin is slim, but the trend lines point toward a shrinking gap between what agents can do and what they need to do.
2026-07-20
Agent evaluation
Sandbox benchmarks are hiding how agents really fail. HKU just built the fix.
UniClawBench evaluates proactive agents across five fundamental capabilities in 400 bilingual real-world tasks, using live Docker containers and a three-agent closed-loop evaluation. It disentangles base model abilities from framework choices, revealing where agents truly break.
2026-07-13
Benchmarks & Tests
Treble Technologies and Hugging Face Launch FFASR Leaderboard for Far-Field Speech Recognition
The new FFASR Leaderboard from Treble Technologies and Hugging Face evaluates ASR models across nine conditions including reverberation, background noise, and microphone distance. Early submissions show far-field WER at low SNR is several times higher than near-field performance, highlighting the need for acoustically robust models.
2026-07-02
Performance
Gemma 4 runs 90% faster in Ollama 0.31 with a trick that needs no config
Ollama 0.31 introduces multi-token prediction for Gemma 4 on Apple Silicon, achieving near 90% faster token generation on coding benchmarks. The speedup comes from an auto-tuned draft model and a custom MLX kernel that eliminates redundant weight reads.
2026-06-29