Inference & Serving · 2023
PagedAttention / vLLM
Manages the key-value cache in non-contiguous fixed-size blocks like virtual-memory paging, removing the fragmentation that wasted most KV-cache memory under contiguous per-request allocation.
Editorial record
Plain-language summary
The KV cache for each sequence is stored in small blocks tracked by a lookup table, so memory is allocated on demand and blocks can be shared across sequences that share a prefix, with copy-on-write on divergence. This cuts memory waste to a few percent and raises the number of requests that can be batched together, and it is the core of the vLLM serving engine. The result was substantially higher serving throughput at the same latency compared to prior systems.
Knowledge graph
Relationships
Descendants
CombinesEvidence: Strongly supported
FlashAttention (1/2/3): IO-Aware Exact Attention
vLLM pairs paged KV with flash attention
P-401
CombinesEvidence: Direct
Orca: Continuous (Iteration-Level) Batching
Continuous batching combined in vLLM
P-401
Source record
Provenance
- Record ID
- P-401
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2309.06180
- arXiv:2309.06180
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.