speculative decoding
4 published articles
LLMs & Inference
Speculative decoding is the nearest free lunch in LLM inference
Speculative decoding accelerates autoregressive generation by proposing candidate tokens with a cheap drafting mechanism and verifying them in a single target-model forward pass, without changing the output. This analysis breaks down the math, the main methods (draft models, EAGLE-3, DFLASH, Multi-Token Prediction, n-gram), and what acceptance rates actually mean for real-world latency.
2026-07-21
Systems optimization
DeepSeek just proved most AI inference speed tricks are working against each other
DeepSeek's DSpark paper reveals that naive speculative decoding degrades throughput under high concurrency. Its solution, confidence-scheduled verification, adapts block length per request and shifts the Pareto frontier of serving performance.
2026-07-12
DeepSeek's DSpark framework rearms speculative decoding for high-concurrency serving
Semi-autoregressive decoding just broke the 85% speed barrier in production AI inference
A new speculative decoding framework from DeepSeek tackles the two bottlenecks that have limited parallel drafters: suffix decay and wasteful verification. DSpark achieves 60, 85% faster generation speeds in production by coupling a semi-autoregressive architecture with a confidence-scheduled scheduler that prunes low-value tokens before the target model verifies them.
2026-07-08
Performance
Gemma 4 runs 90% faster in Ollama 0.31 with a trick that needs no config
Ollama 0.31 introduces multi-token prediction for Gemma 4 on Apple Silicon, achieving near 90% faster token generation on coding benchmarks. The speedup comes from an auto-tuned draft model and a custom MLX kernel that eliminates redundant weight reads.
2026-06-29