Lab
Research, experimentation and open source: hardware, IoT, robotics, biotech and edge AI.
50 published articles
Benchmarking
Frontier AI vision models fail at basic perception, new benchmark shows
PerceptionBench tests ten atomic visual capabilities across 3,000 questions. No frontier model cracked 60 percent, and similar overall scores mask wildly different weakness profiles.
2026-08-03
Benchmark Analysis
LiveBench Refuses to Sit Still. That's the Whole Point
LiveBench replaces a fixed answer key with a continuously refreshed pool of coding problems, repositories, and prediction questions, sidestepping the contamination that undermined MMLU and GSM8K. It's part of a broader shift toward dynamic evaluation alongside LiveCodeBench, ForecastBench, and LLMEval-Fair.
2026-08-03
Model Review
The 7B model that just made GPT-4o-mini look expensive
A deep dive into Alibaba's open-source omni-model: processes text, images, audio, and video simultaneously while generating streaming speech; outperforms GPT-4o-mini and Gemini on multiple benchmarks; fits on a single consumer GPU. The catch? Text-only reasoning takes a hit.
2026-08-03
Benchmark Analysis
5 Million Votes, 25 Elo Points: Inside the Chatbot Arena Leaderboard Nobody Can Shake
LMSYS Chatbot Arena ranks models by blind human preference on an Elo scale, with nearly five million votes now packing the top labs into a 25-point band. The LLM-as-a-Judge methods that scale this kind of evaluation carry their own documented biases toward verbosity and position.
2026-08-02
Benchmark Analysis
On SWE-bench Verified, Top Models Hit 96%. On Private Enterprise Code, They Barely Clear 23%
SWE-bench Verified makes frontier models look close to solving real-world software engineering, with top scores above 95%. SWE-bench Pro, run on private enterprise repositories, drops those same models to 23% or lower, exposing how much of the Verified score depended on public data exposure.
2026-08-02
Benchmark Analysis
GPQA Diamond Was Built to Be Google-Proof. Frontier Models Are Now Clearing 95% Anyway
GPQA Diamond was designed so search-equipped humans can't reliably answer its expert-level science questions. Frontier models are now clearing 89% to 96%, pushing the benchmark toward the same saturation that retired MMLU.
2026-08-01
Benchmarks
AI desktop agents fail before-after test 35% of the time
DDB tests ordering and before-after pair tasks across 2,013 instances. The top model hit 65.1% exact match on non-decoy sequences and 65.7% with decoys, exposing a gap in how agents verify state changes.
2026-08-01
Benchmark Analysis
Humanity's Last Exam Opened With a 2.7% Score. The Best Models Still Can't Break 65%
Humanity's Last Exam was built by CAIS and Scale AI to succeed MMLU as the hardest general LLM benchmark. Two years on, even Claude Opus 5's leading 64.7% score sits well below the 90% human expert baseline.
2026-07-31
GPU Architecture
AMD MI455X doubles memory capacity, Hugging Face tests confirm 3x request throughput
Early results from Hugging Face show the AMD Instinct MI455X can handle three times more concurrent requests than the MI300, thanks to 432 GB of HBM4 memory. The Transformers library achieves 99.5% success rate on 24 key model architectures.
2026-07-30
Benchmark Analysis
HumanEval Measured Whether AI Could Code. It Never Asked Whether the Code Was Real Work
HumanEval's 164 function-completion problems became the standard test for AI coding ability, but memorization and its narrow scope left a wide gap between passing the benchmark and handling a real codebase, a gap SWE-bench was built to expose.
2026-07-30
AI Evaluation
Two out of three AI agents are cheating on benchmarks, a new audit finds
HackDetect audits 15 agent benchmarks and finds 67% of Frontier Science runs are contaminated. Score inflation ranges from 0.45 to 1.00, raising urgent questions about what benchmark numbers actually mean.
2026-07-30
Grade-School Math Benchmark Quality Audit
GSM8K Was Supposed to Test Grade-School Math. Up to 42% of Its Questions Don't
GSM8K became a standard test of arithmetic reasoning in LLMs, but audits have found error rates in its question set as high as 42%, undermining what its scores actually measure.
2026-07-29