Speculative decoding is the nearest free lunch in LLM inference

Most large language models generate text one token at a time. Each new token depends on every one that came before, which means one forward pass per token. On modern GPUs the bottleneck is memory bandwidth, not compute. Every forward pass reads the full set of model weights from memory, and for a single token that is a lot of reading for very little output. Speculative decoding changes the ratio: it lets the model guess several tokens ahead, then verifies the whole batch in one pass. When the guesses are right, the effective throughput goes up. When they are wrong, nothing is lost, because rejection sampling guarantees the output distribution stays identical to standard decoding.

The technique was formalized in 2022 and 2023 by researchers at Google DeepMind, and it has since become a standard feature in local inference engines. The core tradeoff is between the cost of drafting and the acceptance rate. A cheap drafter that guesses poorly can slow things down. An expensive drafter that guesses well can still lose on wall-clock time if the overhead outweighs the saved passes.

The rejection-sampling guarantee

The verification step is what makes speculative decoding mathematically lossless. After a draft model produces a sequence of candidate tokens, the target model evaluates each one against its own conditional distribution. Let p(x) be the probability the target assigns to token x and q(x) the probability the drafter assigned. The token is accepted with probability min(1, p(x) / q(x)). When the target is at least as confident as the drafter, the token always passes. When it is less confident, acceptance scales with the agreement between the two distributions. At the first rejection, verification stops. That token is resampled from the residual distribution, and every subsequent drafted token is discarded because it was conditioned on a prefix that is no longer valid.

The result is that the output is statistically identical to what standard autoregressive decoding would have produced. Identical, trace by trace. The only difference is the number of passes required.

The claim matters because naive implementations can waste the advantage. DeepSeek's DSpark paper showed that under high concurrency, naive speculative decoding can degrade throughput, proving that inference speed tricks can work against each other, similar to the system-level findings in Microsoft's cost-efficiency analysis.

Expected yield and the speedup formula

If each drafted token is accepted with probability α and the drafter proposes up to γ tokens per step, the expected tokens per verification pass is (1 − α^(γ+1)) / (1 − α). For γ = 4 and α = 0.7, that works out to roughly 2.8 tokens per pass, a potential 2.8× throughput increase over standard decoding before accounting for drafting cost. At α = 0.9 the expected yield approaches the full γ+1.

The realized speedup in wall-clock time is lower because drafting is not free. Let c be the cost of one drafting step relative to one target-model forward pass. The formula becomes:

speedup = (1 − α^(γ+1)) / ((1 − α) × (γc + 1))

This is where the different speculative decoding methods diverge. A pure n-gram lookup has nearly zero c but a modest α. A neural drafter like EAGLE-3 has both a higher c and a higher α. Which one wins depends on the workload, the hardware, and the sequence length.

One promising direction is tree-based drafting, which trains a causal draft head over fused hidden states to produce candidate trees that push acceptance rates beyond what linear drafts achieve, a concept that ties into broader work on token-waste reduction.

The main drafting strategies: five approaches in practice

The landscape breaks into two broad families: model-based drafters and pattern-based methods. Model-based drafters use a second neural network to approximate the target model's predictions. The simplest form is a standalone draft model, a much smaller language model trained to imitate the target's distribution. The main cost is that it must be loaded alongside the target model, doubling memory requirements for the drafter alone. EAGLE-3 attaches a lightweight prediction module that operates on the target model's hidden activations, cutting the overhead. DFLASH goes further by using a block-diffusion model that predicts an entire block of future tokens in a single forward pass and then verifies the block as a unit. Multi-Token Prediction (MTP) adds auxiliary prediction heads to the target model itself during training, so the draft comes from the same forward pass as the primary prediction, with no separate model or memory overhead. Pattern-based methods, like n-gram caches and n-gram lookups, exploit repetition already present in the generated text. They cost almost nothing to run because they simply search existing context for matching sequences. In implementations like llama.cpp, the decoder tries pattern-based drafting first, and falls back to a neural drafter only when no suitable continuation is found.

On the hardware side, the Gemma 4's internal representations highlight how open-weight architectures make these drafting techniques especially relevant for self-hosted deployments where memory bandwidth dominates.

What the acceptance numbers actually mean for users

Claims like a 30% to 70% throughput improvement from MTP, or up to 6× from DFLASH, depend heavily on workload characteristics. Predictable continuations, iterating over code, reasoning models that repeat earlier thinking, or summarization that echoes its source, produce long accepted runs and high effective speedup. Short, highly novel, free-form text produces low acceptance rates where the drafting overhead can outweigh the benefit. On Gemma 4, MTP gains can reach roughly 3× under favorable workloads. On Qwen3 27B running on two RTX 5090 GPUs, MTP moves from 51 to 117 tokens per second, about a 2.3× improvement. DFLASH claims even higher peaks, particularly on Qwen3.6, Gemma 4, and Kimi K2.5, but the same dependency on block structure holds: the block size is fixed by the trained draft model, which limits how dynamically the draft length can be adjusted compared with token-by-token methods.

This workload dependency means practitioners need to measure, not assume, as highlighted by cases where benchmarks misled real-world performance: the best strategy depends on the specific inference profile.

The real bottleneck is still memory bandwidth, but the game is changing

Speculative decoding does not reduce the amount of computation the target model performs. It changes the ratio of weight reads to tokens generated. That is why it helps most on memory-bandwidth-bound workloads, which includes most local inference today. As hardware gets faster at moving weights, or as KV cache compression techniques advance, the relative value of speculative decoding may shift. But for the current generation of consumer GPUs and Apple Silicon, it is one of the few free lunches in LLM inference: lower latency, identical output, zero quality cost.

A good concrete case study is the performance of MTP on Gemma 4, where the speedup translates to lower cost per query for self-hosted deployments, a dynamic that reflects broader shifts in who controls AI infrastructure.