The cache contract
For every layer and stored position, preserve the key and value tensors needed by future queries, then append exactly one new pair per decode step.
Reuse computation. Spend memory.Interactive transformer inference simulator
See what a KV cache remembers between tokens
Step through prompt prefill and autoregressive decode. Compare cached keys and values with repeated projection work, then change attention architecture, model shape, precision, and batch size to calculate memory.
With KV cacheReuse prior K and V
Only the new token's projections are appended during decode.
Without cacheRecompute prior K and V
Every decode step repeats projections for the growing prefix.
Current tokens16
KV memory0 MB
Bytes / token0 KB
Projection work ratio1.0x
Attention positions16
Attention architecture
Autoregressive decoding reuses a growing prefix
A decoder-only transformer generates one token at a time. The prompt first passes through a prefill phase that processes many positions in parallel. During decode, each new token creates one query and attends to keys and values from every earlier position.
Without caching, the model would reproduce key and value projections for the complete prefix on every step. A KV cache stores those per-layer tensors after they are computed. The next step appends one new key and one new value per KV head, then uses the stored prefix in attention.
The cache removes repeated projection work, but it does not make decode constant-cost. The new query still compares against a context that grows by one position, cache memory grows linearly with sequence length and batch size, and reading the stored tensors consumes memory bandwidth. Long-context serving is therefore shaped by both arithmetic and memory movement.
The simulator's projection-work ratio is intentionally narrow: it compares how many token positions undergo K/V projection with and without reuse. It is not a latency claim. Kernel fusion, parallelism, batching, hardware, quantization, sampling, and attention implementation all influence wall-clock performance.
The 3D left lane appends one K/V pair on each generated token. The right lane flashes every visible pair because all prefix projections are repeated in the teaching comparison. When the logical sequence exceeds the visible capacity, the scene shows the newest positions while the formulas continue using the full token count.
Prefill and decode differ
Prefill handles the prompt with parallel matrix operations. Decode handles one or a few new positions while reading a large, growing cache.
Memory grows linearly
Double sequence length, batch size, layers, KV heads, head dimension, or bytes per element and the cache grows by the same factor.
Attention still scans context
Caching K and V avoids recomputing projections, but a new query still forms attention scores over prior positions unless a different attention strategy changes that work.
Serving needs allocation policy
Real systems must reserve, page, evict, share, or compress cache blocks while handling requests with different prompt and generation lengths.
Calculate the memory one token adds
Project the hidden state
At each layer, learned matrices map the token representation into query, key, and value vectors. Queries are used for the current attention operation; prior queries are not needed by future tokens in standard decoder self-attention.
Store keys and values by position
The cache keeps K and V for every retained token and layer. Position encoding, layout, quantization metadata, and allocator overhead can add implementation-specific storage beyond the simplified tensor formula.
Read the prefix for the new query
The current query scores the cached keys, masking any disallowed positions. Softmax weights then mix the cached values. Cache hits remove projection repetition, not the need to read and use the prefix.
Append the current K and V
After the step computes the new projections, they become part of the cache for the next token. The sequence dimension grows monotonically unless a window, eviction rule, compression scheme, or request termination changes it.
Manage many requests
Continuous batching interleaves decode steps from requests with different lengths. Paged allocators reduce fragmentation and allow cache blocks to move or share without requiring one large contiguous reservation per sequence.
Change how many key/value heads the cache stores
Multi-head attention
Each query head has its own key and value head. This maximizes head-specific representations but makes cache size proportional to the full attention-head count.
MHAGrouped-query attention
Several query heads share each K/V head. GQA reduces cache memory and projection bandwidth while retaining more K/V groups than a single-head design.
GQAMulti-query attention
All query heads share one key head and one value head. MQA minimizes KV-cache size and bandwidth, trading away per-query-head K/V diversity.
MQACache quantization
Lower-precision cache storage reduces memory and bandwidth. Useful implementations must manage scales, outliers, conversion cost, accuracy, and hardware support.
BitsPrefillDecodeQueryKeyValueBandwidthPagingPrefillDecodeQueryKeyValueBandwidthPaging
Load an inference shape and inspect the tradeoff
Long-context grouped-query model
A long sequence multiplies every per-token cache byte. Grouped-query attention limits the head dimension of that growth, but batch size and layer count can still make the aggregate allocation large.
A KV-cache engineering checklist
Calculate bytes per token from layers, KV heads, head dimension, precision, and batch.
Separate prompt-prefill throughput from decode-token latency.
Measure memory bandwidth and cache reads, not only floating-point operations.
Include allocator metadata, fragmentation, and temporary workspaces in capacity plans.
Test MHA, GQA, or MQA assumptions against the exact model configuration.
Verify cache quantization quality on long and retrieval-heavy prompts.
Define eviction and cancellation behavior for abandoned requests.
Track prefix sharing and isolation boundaries between users or tenants.
Measure continuous-batching behavior across mixed prompt and output lengths.
Monitor cache occupancy, block utilization, preemption, and out-of-memory failures.
Frequently asked questions
Does the KV cache store token text?
No. It stores numerical key and value tensors produced inside each attention layer. Applications may separately retain token IDs, text, logits, or conversation state, but those are not the transformer KV cache itself.
Why are queries not cached?
A past query was used to produce the past token's attention output. Future tokens need their own new query and the prior keys and values. Standard decode therefore caches K and V rather than old Q tensors.
Does caching make each new token constant-time?
No. It removes repeated K/V projection for the prefix, but the new query still attends over a growing sequence and reads a growing cache. Attention kernels, windows, sparsity, and hardware determine the exact scaling.
Can two requests share a KV cache?
Systems can share identical prompt prefixes when identity, position handling, model state, and isolation policy permit it. Prefix caching needs careful correctness, lifecycle, and privacy boundaries.
What happens when the context window is full?
The request may stop, reject additional input, truncate, slide a window, compress memory, or use architecture-specific context management. A cache cannot exceed the positions the model and serving policy support.
Why can cache quantization hurt quality?
Keys influence attention scores and values influence the mixed output. Quantization error accumulates across layers and long contexts, especially for outliers or sensitive heads, so precision choices require evaluation.
Is the projection-work ratio a latency speedup?
No. It counts K/V projection token-work in this teaching comparison. End-to-end latency includes attention, feed-forward layers, memory transfer, synchronization, sampling, and implementation overhead.
Primary research and implementation references
Attention Is All You NeedVaswani and colleagues introduce multi-head scaled dot-product attention and the transformer.
Fast Transformer Decoding: One Write-Head is All You NeedShazeer studies multi-query attention for faster autoregressive decoding.
GQA: Training Generalized Multi-Query Transformer ModelsAinslie and colleagues introduce grouped-query attention as a quality-efficiency tradeoff.
Efficiently Scaling Transformer InferencePope and colleagues analyze transformer inference scaling, partitioning, and memory.
Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon and colleagues describe paged KV-cache allocation for high-throughput serving.
FlashAttentionDao and colleagues present an IO-aware exact attention algorithm.
Three.js DocumentationThe WebGL library used for the synchronized cache visualization.
Super personal AI agentsExplore practical agents that connect model inference with tools and user workflows.
Reuse the prefix, but budget the memory.
KV caching turns repeated projection into persistent state. Efficient inference comes from understanding both sides of that exchange: compute reuse and a cache that grows with every retained token.
Explore text-message AI assistants