Inference & Serving · 2022
Inference Quantization (GPTQ/AWQ/SmoothQuant/bitsandbytes)
Introduced GPTQ, a one-shot post-training weight quantization method that compresses large language model weights to about 3-4 bits using approximate second-order (Hessian) information, letting big models run on far less memory with little accuracy loss.
Editorial record
Plain-language summary
GPTQ quantizes a pretrained model's weights layer by layer after training, using an approximation of the layer's Hessian to decide how to round each weight so that the overall error is minimized, and updating the remaining weights to compensate as it goes. This brings models down to roughly 3-4 bits per weight in one pass without retraining, shrinking memory enough to fit very large models on a single GPU. Related methods such as SmoothQuant (which shifts activation outliers into weights) and AWQ (which protects the weights most important to activations) address the same goal of low-bit inference through different trade-offs.
Knowledge graph
Relationships
Antecedents
EnablesEvidence: Direct
LoRA / QLoRA (parameter-efficient fine-tuning)
4-bit quantization enables QLoRA
P-405
EnablesEvidence: Direct
Edge/Format Ecosystem (llama.cpp/GGUF/TensorRT-LLM/SGLang)
Quantization enables edge/GGUF inference
P-406
Source record
Provenance
- Record ID
- P-404
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2210.17323
- arXiv:2210.17323
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.