gsm8k
2 published articles
LLMs & Models3 min read
AI Research
The monitor that goes silent when AI reasoning fails
Novelis Research shows token log-probability fails as a decoder monitor for quantized reasoning models, being blind to confident loops. They introduce a calibrated e-CUSUM controller that combines uncertainty and repetition signals, achieving selectivity on GSM8K with DeepSeek-R1-Distill-Qwen-1.5B.
2026-08-03
Benchmarks & TestsFeatured2 min read
Grade-School Math Benchmark Quality Audit
GSM8K Was Supposed to Test Grade-School Math. Up to 42% of Its Questions Don't
GSM8K became a standard test of arithmetic reasoning in LLMs, but audits have found error rates in its question set as high as 42%, undermining what its scores actually measure.
2026-07-29