Inference & Serving · 2023
Edge/Format Ecosystem (llama.cpp/GGUF/TensorRT-LLM/SGLang)
This ecosystem built the runtimes and file formats that let LLMs run efficiently on consumer and production hardware, with SGLang contributing RadixAttention, which reuses shared prefix key-value cache across requests via a radix tree to speed up structured and multi-call generation.
Editorial record
Plain-language summary
llama.cpp plus the GGUF format made quantized models runnable on laptops and CPUs; TensorRT-LLM optimizes inference on NVIDIA GPUs; SGLang targets programs that make many related LLM calls. SGLang's RadixAttention keeps a radix tree of previously computed key-value cache so that requests sharing a prompt prefix (few-shot templates, agent loops, branching decodes) skip recomputation, and its language lets developers express constrained, multi-step generations. The result is substantially higher throughput for the structured, repetitive call patterns that agents and serving systems generate.
Knowledge graph
Relationships
Descendants
EnablesEvidence: Direct
Inference Quantization (GPTQ/AWQ/SmoothQuant/bitsandbytes)
Quantization enables edge/GGUF inference
P-406
Source record
Provenance
- Record ID
- P-406
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2312.07104
- arXiv:2312.07104
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.