Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

281reviewed records

22research domains

14 records

0012022Inference & Serving

Peer reviewed

FlashAttention (1/2/3): IO-Aware Exact Attention

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, Jay Shah

The kernel splits queries, keys, and values into blocks that fit in fast on-chip SRAM, computes softmax incrementally with running statistics, and recomputes intermediates during the backward pass instead of storing them, so memory scales linearly rather than quadratically in sequence length. This yields wall-clock speedups and lower memory use without approximation, giving identical results to standard attention. It enabled training and serving with longer context windows and became a default attention implementation in mainstream frameworks.

UnknownDifficulty 6/10Verified
Read record Primary source
0022023Inference & Serving

Peer reviewed

PagedAttention / vLLM

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Ion Stoica

The KV cache for each sequence is stored in small blocks tracked by a lookup table, so memory is allocated on demand and blocks can be shared across sequences that share a prefix, with copy-on-write on divergence. This cuts memory waste to a few percent and raises the number of requests that can be batched together, and it is the core of the vLLM serving engine. The result was substantially higher serving throughput at the same latency compared to prior systems.

UnknownDifficulty 5/10Verified
Read record Primary source
0032022Inference & Serving

Peer reviewed

LoRA / QLoRA (parameter-efficient fine-tuning)

Edward J. Hu, Yelong Shen, Weizhu Chen, Tim Dettmers, Luke Zettlemoyer

Instead of updating a weight matrix directly, the method learns a low-rank product added to it, so only a tiny fraction of parameters are trained and the small adapters can be stored per task and swapped in. Because the update can be merged back into the base weights after training, there is no extra latency at inference, unlike adapter layers. On GPT-3 175B it reduced trainable parameters by roughly ten-thousand-fold while matching full fine-tuning quality, making per-task customization of large models cheap and modular.

UnknownDifficulty 5/10Verified
Read record Primary source
0042015Inference & Serving

Preprint

Distilling the Knowledge in a Neural Network

Geoffrey Hinton, Oriol Vinyals, Jeff Dean

Instead of training only on hard labels, the student is trained on the teacher’s full probability distribution softened by a temperature, so it inherits the relative confidences the teacher assigns across classes — information the hard label throws away. Students recover much of the teacher’s performance at a fraction of the parameters and compute. Distillation became a standard compression tool and, later, the way strong small language models are trained from frontier teachers.

UnknownDifficulty 5/10Verified
Read record Primary source
0052022Inference & Serving

Peer reviewed

Orca: Continuous (Iteration-Level) Batching

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, et al.

Traditional batching runs a whole batch of requests together until all finish, so short requests wait for long ones and the GPU sits underused. Orca schedules at the granularity of a single decoding iteration: after each token step it can inject newly arrived requests and remove completed ones, keeping the batch continuously full. Combined with selective batching that handles operations with different sequence lengths correctly, this delivered large throughput and latency gains and became the standard serving pattern later adopted by systems like vLLM.

UnknownDifficulty 5/10Verified
Read record Primary source
0062023Inference & Serving

Peer reviewed

Speculative Decoding / Medusa / EAGLE

Yaniv Leviathan, Yossi Matias, Charlie Chen, Tianle Cai, Tri Dao, Yuhui Li, Hongyang Zhang

Autoregressive generation is slow because each token requires a full forward pass of the large model, run one at a time. Speculative decoding runs a cheap draft model to guess a short run of upcoming tokens, then the large model checks all of them in a single parallel pass and accepts the longest correct prefix, resampling at the first mismatch. A modified acceptance rule guarantees the result matches what the large model would have produced on its own, so it speeds up inference (often 2-3x) with no loss in output quality.

UnknownDifficulty 6/10Verified
Read record Primary source
0072022Inference & Serving

Peer reviewed

Inference Quantization (GPTQ/AWQ/SmoothQuant/bitsandbytes)

Elias Frantar, Dan Alistarh, Ji Lin, Song Han, Guangxuan Xiao, Tim Dettmers, Luke Zettlemoyer

GPTQ quantizes a pretrained model's weights layer by layer after training, using an approximation of the layer's Hessian to decide how to round each weight so that the overall error is minimized, and updating the remaining weights to compensate as it goes. This brings models down to roughly 3-4 bits per weight in one pass without retraining, shrinking memory enough to fit very large models on a single GPU. Related methods such as SmoothQuant (which shifts activation outliers into weights) and AWQ (which protects the weights most important to activations) address the same goal of low-bit inference through different trade-offs.

UnknownDifficulty 6/10Verified
Read record Primary source
0082023Inference & Serving

Software release

Edge/Format Ecosystem (llama.cpp/GGUF/TensorRT-LLM/SGLang)

Georgi Gerganov, Lianmin Zheng, Ying Sheng, Ion Stoica, NVIDIA

llama.cpp plus the GGUF format made quantized models runnable on laptops and CPUs; TensorRT-LLM optimizes inference on NVIDIA GPUs; SGLang targets programs that make many related LLM calls. SGLang's RadixAttention keeps a radix tree of previously computed key-value cache so that requests sharing a prompt prefix (few-shot templates, agent loops, branching decodes) skip recomputation, and its language lets developers express constrained, multi-step generations. The result is substantially higher throughput for the structured, repetitive call patterns that agents and serving systems generate.

Source availableDifficulty 5/10Verified
Read record Primary source
0092025Inference & Serving

Preprint

BitNet b1.58 2B4T Technical Report

Shuming Ma, Hongyu Wang, Shaohan Huang, et al. (Microsoft Research)

First open 2B-scale LLM trained natively in 1.58-bit ternary weights, matching full-precision peers of its size while cutting memory ~6x and running fast on CPU.

UnknownDifficulty 6/10Verified
Read record Primary source
0112025Inference & Serving

Technical report

Defeating Nondeterminism in LLM Inference

Horace He

Traces run-to-run nondeterminism in LLM serving to batch-size-dependent kernel reductions rather than float non-associativity, and gives batch-invariant RMSNorm/matmul/attention kernels that make inference bitwise-reproducible.

UnknownDifficulty 5/10Verified
Read record Primary source
0122025Inference & Serving

Technical report

LoRA Without Regret

John Schulman

Shows that LoRA applied to all layers with the right hyperparameters (notably a learning rate ~10x higher than full fine-tuning) matches full fine-tuning's learning efficiency and final quality across typical post-training regimes.

UnknownDifficulty 5/10Verified
Read record Primary source
0132025Inference & Serving

Technical report

On-Policy Distillation

Kevin Lu

Combines on-policy sampling from the student with dense per-token teacher supervision, reaching RL-level reasoning-benchmark performance at a fraction of RL's compute cost.

UnknownDifficulty 5/10Verified
Read record Primary source
0142025Inference & Serving

Product release

Announcing Tinker

Thinking Machines Lab

A managed fine-tuning API exposing low-level primitives (forward_backward, sample) so users write post-training loops in Python while Tinker handles distributed GPU execution, including for large MoE open models like Qwen-235B-A22B.

UnknownDifficulty 5/10Verified
Read record Primary source