Lab
Research, experimentation and open source: hardware, IoT, robotics, biotech and edge AI.
50 published articles
Benchmark Analysis
MMLU Ruled AI Benchmarking for Years. Then Models Started Acing the Answer Key
MMLU defined a generation of AI benchmarking, but training-data contamination and a flawed answer key have pushed frontier labs toward dynamic successors like HLE and GPQA Diamond.
2026-07-28
KlingTeam
Video generation's dirty secret is finally on the record
KlingTeam introduces MultiRef-Compass and KeyFrame-Compass, two benchmarks that push video generation models beyond text-to-video and into multi-reference and keyframe-conditioned tasks. Early tests on eight and nine systems respectively show consistent failures in binding entities, preserving temporal order, and handling dense constraints.
2026-07-26
Hardware Inflation
The memory shortage is making your next gadget more expensive
Qualcomm and Roku lead a wave of hardware price hikes as a memory shortage, fueled by AI demand, is expected to last through 2027. The increases ripple across smartphones, streaming devices, and other electronics, with no relief in sight.
2026-07-26
Hardware
A 128 GB desktop that undercuts Nvidia cloud rentals by 6x
AMD's Ryzen AI Halo targets local AI development with 128 GB unified memory, support for up to 200B parameter models, and claims of up to 7.3x faster performance than Apple M4 Pro on certain image generation tasks. The platform runs on both Windows and Linux, a differentiator against Nvidia's Linux-only DGX Spark.
2026-07-25
Benchmark
LLMs can describe data. They cannot reason through it. A new benchmark proves the gap is real.
SDABench, a new capability-oriented benchmark spanning six core scientific reasoning skills and five domains, tests 15 LLMs and finds that models are strong on descriptive analysis but collapse on inferential and causal tasks. The paper provides a five-stage error analysis framework to localize failures.
2026-07-25
Benchmark & Tests
Your graph model breaks on dirty data. A new benchmark shows exactly where.
OpenRTAG, a benchmark from an academic team, organizes TAG quality issues into a 3×3 taxonomy covering text, structure, and label degradation. It tests traditional GNNs, large language model-enhanced GNNs, and graph foundation models across nine datasets, revealing different sensitivity patterns that prior fragmented studies missed.
2026-07-24
Agent evaluation
Your AI agent keeps failing? It might be the harness, not the brain
PawBench, an open-source benchmark from the AgentScope team, systematically evaluates models and agent harnesses together. Results show that harness design can swing scores by over 11 points for smaller models, exposing a blind spot in how AI agents are currently judged.
2026-07-22
Sakana AI's collective intelligence finally goes physical
These identical bricks taught themselves to recognize their own shape. No central brain needed.
A stack of identical cubes, each running the same tiny neural network and talking only to its immediate neighbors, can figure out their global shape, detect damage, and guide regrowth. No single brick knows where it is. The system hit 100 percent accuracy in hardware, published in Nature Communications.
2026-07-21
Agentic coding
Grok 4.5 just broke the coding agent leaderboard: the lead is real, the margins are tiny
Grok 4.5 now leads the SWE Marathon leaderboard, beating Claude 4 Opus and GPT-5. The benchmark tests real software engineering skills: bug fixes, feature additions, and code understanding across real repositories. The margin is slim, but the trend lines point toward a shrinking gap between what agents can do and what they need to do.
2026-07-20
AI Research
BMW's 16 GB GPU just did what needs an A100
BMW researchers show that Hierarchical Global Attention, paired with truncated backprop and external KV storage, lets a 16 GB GPU train on 16K tokens, four times the limit of dense attention, with no measurable loss in adapter quality.
2026-07-20
NVIDIA
Nvidia and Hugging Face just made distributed diffusion training boring (that's the point)
Nvidia's NeMo Automodel now integrates directly with Hugging Face Diffusers, enabling production-grade distributed training for models like FLUX, Wan 2.1, and HunyuanVideo. The Apache 2.0 library handles parallelism as a config toggle and lets fine-tuned checkpoints load straight back into inference pipelines.
2026-07-19
AI & Robotics
Nvidia's robot model just unlocked a three-order-of-magnitude advantage: time
NVIDIA Research's RoboTTT scales robot model context to 8K timesteps, three orders of magnitude beyond current policies, unlocking one-shot imitation from human video and 87% performance gains, suggesting context length as a new scaling axis for robot foundation models.
2026-07-17