What LLM prefix caching can reuse

The work before the first token

Picture a support assistant that sends the same instructions and product handbook with every question. A decoder-only language model first processes the prompt in a stage called prefill. At each layer it computes key and value vectors for every prompt token. A serving engine can store those vectors, so a later request with the same opening tokens starts from the end of the cached span. vLLM calls this automatic prefix caching, and hosted APIs from Anthropic and OpenAI offer prompt caching on the same principle.

Generation still runs normally. At each decoding step the model computes a new query and attends to the stored keys and values, including those of the tokens it has already generated. The cache holds intermediate states from earlier positions. The query and the attention output for the next position still have to be computed.

A small edit can invalidate a long suffix

Take a toy sequence A B C D E F. A later request that starts A B C D E F G can reuse all six cached tokens and compute only G. If the request starts A B C X E F, the unchanged prefix ends at C. In a standard causal transformer with full attention, the hidden state at E or F can depend on the edited token, so matching text further down the prompt does not bring the cached states back. Anthropic describes the same rule for its API: changes at one level of the prompt invalidate that level and everything after it. OpenAI recommends putting stable instructions and shared reference material first and dynamic content last.

Engines also cache in blocks. With a block size of two tokens, the example splits into A B, C D and E F. For the edited request only A B is reused. C is unchanged but shares a block with X, so that block misses and the blocks after it miss too. Block size is a setting, so the exact boundary varies by engine.

What the cache key must include

Cache matching happens after tokenization and depends on the model’s effective input. Positions, model and adapter identity, and any input that changes activations all matter. vLLM hashes each cached block together with the hash of the block before it, and adds extra values such as LoRA IDs, multimodal input hashes and cache salts. A block is therefore identified by everything that precedes it. The PagedAttention paper describes the block-based KV cache this builds on, and the SGLang paper describes RadixAttention, which keeps cached prefixes in a tree so that requests share their common opening tokens.

Compare cached and uncached outputs at the same precision. Kernel ordering and rounding can produce small numerical differences even when both paths compute the same mathematical result.

Prefix caching and exact attention solve different costs

A prefix hit saves the work of processing a prefix again. Each new token still attends to the permitted context under the model’s attention mask. For a full-attention model, FlashAttention is a kernel that computes exact attention while reducing transfers between GPU memory and faster on-chip storage. A serving stack can use it on cache misses and for the work that remains after a hit. Sparse attention or dropping context changes which inputs enter the calculation and calls for a separate quality test.

Measure hits, misses, and decode separately

Replay the same request mix with prefix caching off and on. Include cold requests, repeated long prefixes, and prompts edited near the start. Keep the model, hardware, request order, concurrency and output lengths comparable. Report time to first token and prefill time, then report time per generated token separately. Record how many prompt tokens hit the cache and how much cache memory the service used. If energy matters, meter joules per completed request under the same traffic pattern. The numbers show which reuse survives real traffic and eviction pressure.

Sources

• Vaswani et al., Attention Is All You Need
• vLLM design documentation for automatic prefix caching
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs
• Anthropic documentation on prompt caching
• OpenAI documentation on prompt caching
• Dao et al., FlashAttention