Speculative decoding is the nearest free lunch in LLM inference

Most large language models generate text one token at a time. Each new token is predicted from every one that came before it, which means one forward pass per token. On modern GPUs the bottleneck is rarely compute, it is memory bandwidth. Every forward pass reads the full model's weights from memory, and for a single token that is a lot of reading for very little writing. Speculative decoding changes the ratio: it lets the model guess several tokens ahead, then verifies the whole batch in one pass. When the guesses are right, the effective throughput goes up. When they are wrong, nothing is lost, the rejection sampling guarantees the output distribution stays identical to standard decoding.

The technique was formalized in 2022, 2023 by researchers at Google DeepMind, and it has since become a standard feature in local inference engines. The core tradeoff is between the cost of drafting and the acceptance rate. A cheap drafter that guesses poorly can actually slow things down. An expensive drafter that guesses brilliantly can still lose on wall-clock time if the overhead outweighs the saved passes.

The rejection-sampling guarantee

The verification step is what makes speculative decoding mathematically lossless. After a draft model produces a sequence of candidate tokens, the target model evaluates each one against its own conditional distribution. Let p(x) be the probability the target assigns to token x and q(x) the probability the drafter assigned. The token is accepted with probability min(1, p(x) / q(x)). When the target is at least as confident as the drafter, the token always passes. When it is less confident, acceptance scales with the agreement between the two distributions. At the first rejection, verification stops. That token is resampled from the residual distribution, and every subsequent drafted token is discarded because it was conditioned on a prefix that is no longer valid.

The result: the output is statistically identical to what standard autoregressive decoding would have produced. Not approximately identical, identical, trace by trace. The only difference is the number of passes required to get there.

The claim matters because naive implementations can waste the advantage. DeepSeek's DSpark paper showed that under high concurrency, naive speculative decoding can degrade throughput, proving most inference speed tricks can actually work against each other, as detailed in DeepSeek's own analysis.

Expected yield and the speedup formula

If each drafted token is accepted with probability α and the drafter proposes up to γ tokens per step, the expected tokens per verification pass is (1 − α^(γ+1)) / (1 − α). For γ = 4 and α = 0.7, that works out to roughly 2.8 tokens per pass, a potential 2.8× throughput increase over standard decoding before accounting for drafting cost. At α = 0.9 the expected yield approaches the full γ+1.

The realized speedup in wall-clock time is lower because drafting is not free. Let c be the cost of one drafting step relative to one target-model forward pass. The formula becomes:

speedup = (1 − α^(γ+1)) / ((1 − α) × (γc + 1))

This is where the different speculative decoding methods diverge. A pure n-gram lookup has almost zero c but a modest α. A neural drafter like EAGLE-3 has a higher c but a higher α. Which one wins depends on the workload, the hardware, and the sequence length.

One promising direction comes from JetSpec's parallel tree drafting, which trains a causal draft head over fused hidden states, producing candidate trees that push acceptance rates far beyond what linear drafts achieve.

The main drafting strategies: five approaches in practice

The landscape breaks into two broad families: model-based drafters and pattern-based methods. Model-based drafters use a second neural network to approximate the target model's predictions. The simplest form is a standalone draft model, a much smaller language model trained to imitate the target's distribution. The main cost is that it must be loaded alongside the target model, doubling memory requirements for the drafter alone. EAGLE-3 attaches a lightweight prediction module that operates on the target model's hidden activations, cutting the overhead significantly. DFLASH goes further by using a block-diffusion model that predicts an entire block of future tokens in a single forward pass, then verifies the block as a unit. Multi-Token Prediction (MTP) adds auxiliary prediction heads to the target model itself during training, so the draft comes from the same forward pass as the primary prediction, no separate model or memory overhead. Pattern-based methods, like n-gram caches and n-gram lookups, exploit repetition already present in the generated text. They cost almost nothing to run because they simply search existing context for matching sequences. In implementations like llama.cpp, the decoder tries pattern-based drafting first, and falls back to a neural drafter only when no suitable continuation is found.

On the hardware side, Gemma 4's open-weight architecture makes these drafting techniques especially relevant for self-hosted deployments where memory bandwidth is the dominant constraint.

What the acceptance numbers actually mean for users

Claims like a 30, 70% throughput improvement from MTP, or up to 6× from DFLASH, depend heavily on workload characteristics. Predictable continuations, iterating over code, reasoning models that repeat earlier thinking, or summarization that echoes its source, produce long accepted runs and high effective speedup. Short, highly novel, free-form text produces low acceptance rates where the drafting overhead can outweigh the benefit. On Gemma 4, MTP gains can reach roughly 3× under favorable workloads. On Qwen3 27B running on two RTX 5090 GPUs, MTP moves from 51 to 117 tokens per second, about a 2.3× improvement. DFLASH claims even higher peaks, particularly on Qwen3.6, Gemma 4, and Kimi K2.5, but the same dependency on block structure holds: the block size is fixed by the trained draft model, which limits how dynamically the draft length can be adjusted compared with token-by-token methods.

This workload dependency means practitioners need to measure, not assume, as Sakana AI's model routing approach demonstrates: the best strategy depends on the specific inference profile.

The real bottleneck is still memory bandwidth, but the game is changing

Speculative decoding does not reduce the amount of computation the target model performs. It changes the ratio of weight reads to tokens generated. That is why it helps most on memory-bandwidth-bound workloads, which is most local inference today. As hardware gets faster at moving weights, or as KV cache compression techniques advance, the relative value of speculative decoding may shift. But for the current generation of consumer GPUs and Apple Silicon, it is one of the few free lunches in LLM inference: lower latency, identical output, zero quality cost.

A good concrete case study is the performance of MTP on Gemma 4, also covered in the analysis of Gemma 4 as infrastructure, where the speedup directly translates to lower cost per query for self-hosted deployments.