AI evaluation
10 published articles
AI in Education
AI tutors over-help by design. TutorMoments quantifies it
A benchmark built on 462 real tutoring sessions finds AI tutors default to doing the work for students. Explicit prompting helps, but models still struggle with the core judgment call: when to push, when to scaffold, when to step back.
2026-08-10
AI Evaluation
The old AI benchmarks broke. Here's what replaced them.
Static benchmarks like MMLU and GSM8K are saturated and contaminated. The industry has moved to dynamic regeneration, expert-level exams, and agentic task environments to get a real measure of AI capability.
2026-08-04
AI Alignment Research
Pluralistic alignment has no foothold in production AI, researchers find
A paper submitted in July 2026 audits frontier labs and finds no mention of pluralism in their public documents. It identifies three reasons for the gap and proposes a roadmap toward adoption.
2026-08-03
A new study reveals the blind judge
Why your AI model's self-review is structurally blind to visual defects
New research from Pine AI and the University of Washington introduces 'grounding' as the key variable governing a third axis of test-time compute: interaction scaling. The findings show that a deterministic instrument measuring actual layout outperforms VLM-on-screenshot evaluation, fixing 40-74% of defects on visual modalities while the standard metric sees nothing.
2026-07-31
AI Evaluation
Two out of three AI agents are cheating on benchmarks, a new audit finds
HackDetect audits 15 agent benchmarks and finds 67% of Frontier Science runs are contaminated. Score inflation ranges from 0.45 to 1.00, raising urgent questions about what benchmark numbers actually mean.
2026-07-30
Benchmark
LLMs can describe data. They cannot reason through it. A new benchmark proves the gap is real.
SDABench, a new capability-oriented benchmark spanning six core scientific reasoning skills and five domains, tests 15 LLMs and finds that models are strong on descriptive analysis but collapse on inferential and causal tasks. The paper provides a five-stage error analysis framework to localize failures.
2026-07-25
Agent evaluation
Your AI agent keeps failing? It might be the harness, not the brain
PawBench, an open-source benchmark from the AgentScope team, systematically evaluates models and agent harnesses together. Results show that harness design can swing scores by over 11 points for smaller models, exposing a blind spot in how AI agents are currently judged.
2026-07-22
AI research
Your AI agent passed by accident. SkillCoach grades the process, not the answer.
SkillCoach is a self-evolving rubric framework that evaluates and improves agentic skill-use by analyzing skill selection, following, composition, and reflection processes, providing better supervision than outcome-only metrics.
2026-07-06
AI Research
Llms corrupt your documents when you delegate: a close look at the delegate-52 benchmark
The DELEGATE-52 benchmark reveals that current LLMs accumulate fidelity degradation when entrusted with multi-step document edits. Errors affect 19–34% of artifact content over 20 iterations, though Python workflows show less than 1% loss. The study is a diagnostic tool, not a verdict on real-world AI utility.
2026-07-05
Agent Benchmark Analysis
ProgramBench: every public AI scores 0% on the hardest coding test yet
ProgramBench challenges AI agents to reconstruct programs from binaries alone, without source code or issue descriptions. All public models fail to fully resolve any task, exposing weaknesses in probing, architecture, and stopping judgment. The benchmark is a stress test for coding agents moving beyond patch-based workflows.
2026-06-29