One routing pass to rule them all: 7x faster LLM decoding without the quality trade-off

Attention is still the bottleneck. Every extra token of context in a transformer means more keys and values to store and sift through, and decoding a long chain-of-thought can stall even on high-end hardware. Sparse attention tries to fix this by ignoring the irrelevant parts of the cache, but the fix often costs accuracy (block sparse) or speed (token sparse, where picking the right tokens each layer is its own bottleneck).

A new paper, CLSA: You Only Index Once, argues that both problems have a single root cause: each layer runs its own routing independently. The authors propose sharing the routing index across layers in a KV-cache-sharing architecture, so the expensive top-k selection runs once and the result is reused. The experiments report up to 7.6x faster decoding and 17.1x better overall throughput at 128K tokens, with accuracy matching full attention on standard benchmarks. This is one of those rare architectural fixes where efficiency and quality improve together, not trade places.

How CLSA works

The method builds on hybrid attention architectures that share KV caches across cross-decoder layers, the YOCO family, for instance. In those models the KV cache is already reused, but each cross-decoder layer still runs its own top-k routing over the full cache. CLSA adds one shared indexer layer: it computes the attention scores once, picks the top-k tokens, and passes that same index to every subsequent layer.

That single pass replaces N routing passes, where N is the number of cross-decoder layers. The routing itself becomes a fixed cost instead of a linear one in the number of layers, which matters more as models grow deeper and contexts grow longer.

The key move is keeping token-level sparsity, the fine-grained selection that makes accuracy high, while amortizing the overhead that normally makes it slow. Block sparse methods skip whole memory pages for speed but miss relevant tokens inside the skipped blocks. CLSA avoids that by keeping per-token granularity and paying for it exactly once.

Where the speed comes from

The paper measures three inference stages: pre-filling (processing the prompt), KV-cache storage (the memory footprint of the cached keys and values), and autoregressive decoding (generating tokens one by one). All three improve simultaneously.

These numbers come from a model with a 128K-token context window. The decoding speedup alone, 7.6x, means a generation that takes 8 seconds with full attention finishes in just over 1 second with CLSA. Throughput gains are even larger because the model can batch more requests with the reduced memory footprint. Compare that to the kind of thinking suggested by BMW's pragmatic approach to attention on consumer GPUs, and the step change here is obvious.

Accuracy holds up

On the RULER long-context benchmark (averaged over five prompting templates and multiple lengths), CLSA matches full attention within 0.5 points. On short-context tasks, MMLU, ARC-Challenge, HellaSwag, the difference is below 0.3 points. That is essentially noise. On GSM8K (math reasoning) the gap widens to about 1.2 points, which the authors attribute to the fact that small token selections can miss critical numbers in arithmetic chains.

Still, the trade-off is unusual: a method that delivers 7x speed gain while losing less than 1.5 points on any benchmark is rare among sparse attention proposals. Block sparse methods typically lose 3, 5 points on the same tasks for comparable speed.

The accuracy stability comes from the fact that the routing index is computed once using attention scores from the indexer layer, which sees the same query and cache state as every other layer, the top-k set for one layer is a strong proxy for the top-k set of all layers in a shared-cache architecture.

The cost: compatibility limits

CLSA is not a drop-in replacement for any transformer. It requires a KV-cache-sharing backbone (YOCO, or models built with cross-decoder layers) to work. A standard decoder-only transformer that does not share caches across layers would not benefit from shared routing, because each layer's KV cache is different and the index would point to stale entries.

Practitioners running speculative decoding for long-context reasoning may find CLSA complementary, the draft model could use the same shared routing for speed, while the target model runs full attention for verification. The paper does not explore this combination, but the architectures are compatible.

The other limitation is the fixed top-k selection count. The paper uses k=512 tokens out of a 128K cache, about 0.4% density. For tasks requiring very dense attention, e.g., scanning every token of a financial contract for a compliance check, such aggressive sparsity may miss detail even if the routing index is accurate. The authors suggest the indexer layer can adjust k dynamically, but this is left as future work.

What it means for long-context inference

The paper's strongest claim is that the efficiency-accuracy trade-off is not inevitable, it is an artifact of routing independently per layer. A single shared routing pass solves both sides: accuracy stays high because selection is token-level, and speed improves because routing cost is amortized across layers.

At 128K context, the 17x throughput gain means a single GPU can serve roughly 17x more concurrent long-context requests than with full attention. For cloud inference providers, that directly cuts per-token cost. For on-device open-weight models, it reduces the memory and latency gap between running short and long prompts.

There are still open questions: how CLSA behaves beyond 128K (the paper tests up to that point); whether dynamic k selection can close the <1.5-point gap on arithmetic reasoning tasks; and how well the method scales to 70B+ parameter models where the routing index itself becomes a large data structure. Prior work on self-sabotaging optimization loops suggests that scaling up a technique can introduce surprises, so this one bears watching. But as a solution to the specific bottleneck that makes long-context inference expensive, it addresses the right bottleneck in a principled way.