LLM Infrastructure
Why LLM memory gets expensive.
Long prompts are not expensive only because they contain more text. They are expensive because the model has to keep reusable attention state available while the response is being generated.
That cache is separate from the model's learned knowledge. It keeps the attention state for tokens the model has already processed, so generation can continue without rebuilding earlier work.
Practical Takeaway
Long context is a memory bandwidth problem, not just a storage problem.
During decoding, the model reads cached state over and over as it emits tokens. Bigger context, larger batches, and more concurrent users all increase that pressure.
Working Memory
The KV cache prevents recomputation and creates a new bottleneck.
A transformer extends a response step by step. Each fresh token needs information from earlier tokens. Without cached attention state, the model would repeat old work on every step, burning compute for no useful reason.
The KV cache stores those vectors after they are computed. The next token can reuse them. The speed win is real, but now the serving system has to hold and read a large cache for every active request.
Inference Phases
Prefill and decode stress the hardware differently.
Prefill reads the whole input and builds the initial cache. It is compute-heavy and parallel. The GPU can process many input tokens together.
Decode produces the answer incrementally. It is memory-heavy because every generated token has to consult cached state. A request can fit in memory and still be slow if cache movement is the limiting factor.
kv_cache_bytes = 2
* layers
* key_value_heads
* head_dimension
* bytes_per_value
* context_tokens
* batch_sizeReduction Techniques
Every optimization attacks one term in the equation.
Grouped-query attention reduces the number of key-value heads stored for each layer. Multi-query attention goes further, but often with stronger quality tradeoffs.
Quantization stores cached values with smaller numeric formats. Eight-bit cache storage is often practical; more aggressive compression can save additional memory while hurting demanding retrieval tasks.
Eviction drops tokens that appear less useful, usually keeping recent tokens and important early anchors. It can work well for casual chat and fail badly when a later answer needs a fact from the middle of the context.
Grouped-query attention
Shrink what each token stores.
Quantization
Store each value with fewer bits.
Eviction
Keep fewer tokens, with retrieval risk.
Serving Systems
Serving engines reduce waste around cached state.
Paged attention manages cache memory like an operating system manages memory pages. Instead of reserving one large block for each request, the system allocates smaller blocks as needed. That reduces fragmentation.
Prefix caching lets requests with the same beginning share computed state. This matters for agents because the same instructions and tool schemas often appear at the front of every call.
Product Choices
Most product teams reduce memory pressure before touching model internals.
If you are building an app on hosted models, you probably cannot change grouped-query attention or latent attention. You can still control the prompts you send, how much history you retain, which files enter context, and whether repeated prefixes stay stable enough for caching.
For RAG systems, the answer is usually better retrieval, not a larger prompt. Send the model the smallest set of passages that can answer the question and keep provenance attached so the answer can be checked.
For agents, compact old tool results, summarize completed steps, avoid resending unused tool schemas, and store durable state outside the model. The model should receive the slice of memory needed for the next decision, not the whole project archive.
Shorten context deliberately
Pruning is an architecture decision, not just a token-saving trick.
Separate state from prompt
Databases, logs, and files should hold durable truth.
Cache stable prefixes
Repeated system instructions and schemas should stay byte-stable.
Related Reading