AI tutors over-help by design. TutorMoments quantifies it

A math model that solves problems is not the same as a math model that teaches. The gap sits in a judgment call that recurs every session: does this student need support right now, or a push to do the reasoning themselves? The team behind TutorMoments, a benchmark built from 462 real one-on-one math tutoring transcripts, introduced it this week with a blunt finding: left to their default instincts, language models over-help.

Why tutor benchmarks have missed the moment so far

Most existing evaluations of AI tutors reward one fixed behavior, whether that means never giving the answer or always offering a hint. The problem, as the TutorMoments authors frame it, is that neither is right in every situation. Good tutoring is a judgment call about what this student needs, right now, on this problem, and a fixed rubric cannot see the student, the same flaw that broke the static benchmarks.

The blind spot is showing up across the field. K12-Bench, a Peking University evaluation of how well models understand school math, found even strong models barely grasp how concepts connect, scoring 57% and 46% on items built around prerequisite chains and concept taxonomies, the connective tissue a real tutor uses daily. SDABench, which tests 15 models across six scientific reasoning skills, found them strong on descriptive questions but collapsing on inferential and causal ones, per the SDABench findings. The pattern: models get better at knowing things while the harder skill, deciding what to do with that knowledge in a specific situation, stays unmeasured.

Meanwhile, the stakes got higher. A Dartmouth College course where an AI tutor posted effect sizes up to 1.30 standard deviations delivered learning gains rarely seen in a live classroom, where typical interventions hover around 0.4 SD. The more confidence the field puts in AI tutoring, the more it matters that the moment-by-moment judgment call is exactly what benchmarks have not been measuring.

Told to "tutor well", models rush to help

TutorMoments-Preview consists of 462 de-identified, text-only transcripts from a high-dosage U.S. tutoring program serving mostly Title I schools, with students in grades 2-7. Twenty-seven teacher annotators flagged more than 1,500 key moments, each one a decision point where the tutor had to weigh scaffolding (making the problem more accessible) against pushing for rigor (asking for harder thinking).

The evaluation pauses a transcript at one of those moments and hands the session to a language model, which tutors a simulated student for five turns. A scoring pipeline, starting from the teachers' majority label for each moment, checks whether the model scaffolded when support was called for, pushed for rigor when the student was ready, and avoided over-scaffolding.

Seven models ran the setup twice: once with a plain prompt telling them to use what they know about good tutoring, once with the trade-off spelled out. The clearest pattern in the results is how much the prompt matters. Every model scores higher under the explicit prompt, which suggests a model's default "helpful assistant" behavior is not enough on its own to tutor well. But spelling out the trade-off only goes so far. Models still differ widely in how reliably they make the call, and the gap to human tutoring that consistently fits the moment does not close. When they did push for rigor, the models relied on a narrow set of moves, mostly asking students to explain their answers, while human tutors used more varied strategies and were much more likely to step back and let the student work independently.

The human reference scores lower. That's not the headline

Here is the part that will get quoted out of context. Scored the same way at the same decision points, the human tutors in the transcripts land below the models: 0.458 for appropriate scaffolding, 0.182 for appropriate rigor, 0.496 for avoiding over-scaffolding, all out of 1. The model scores under the evaluation-aware prompt sit above those bars.

The authors are explicit that this is not a claim that AI tutors outperform human teachers. The annotators looked for moments where tutoring could have gone better, so the dataset concentrates on missed opportunities, not ideal practice. Two more caveats: the scores measure tutor behavior, not learning, since the replays run against a simulated "oracle" student; and rigor is the noisier signal, with fewer annotated rigor moments (260 versus 738) and a scoring pipeline that detects pushes for rigor less reliably.

What the edtech market should take from this

For product teams, the headline lever is prompt design. Spelling out the trade-off lifts every score, which says the out-of-box helpfulness of a general assistant is the wrong default for tutoring. The same result warns that a better prompt is not a fix: K12-Bench's finding sharpens the warning, because a tutor that does not track which concepts are prerequisites for which will struggle with the exact decision TutorMoments measures.

The commercial context is real. Khanmigo, built on GPT-4, has shown promise but published limited controlled efficacy data, while Duolingo's AI lessons and startups such as Querium and Photomath have shipped tutoring features with variable results. A benchmark that can tell an AI tutor when it is doing the work for the student is one the whole category needs, even if it only measures behavior at a decision point. The authors draw that line themselves: automated evaluation cannot stand in for studies with real students and real learning outcomes. Agent benchmarks are hitting the same wall, which is why sandbox tests are losing credibility. The dataset is narrow, U.S.-based, mostly elementary and middle-school math, annotated by a single pool of educators. They are building toward a larger multimodal dataset and a stronger scoring pipeline, and they released the transcripts, the replay code, and the model replays so anyone can rerun the evaluation.

For now the takeaway is fairly clean. The field finally has an instrument that watches a tutor make the call, and the first readings say the models can do the math but miss the teaching, the same split HumanEval left unexamined for coding. The next benchmark worth watching will follow real students, because a judgment call about how much help a learner needs can only be validated where learning actually happens.