Open-Weight Model Families · 2023
Mistral 7B / Mixtral
A compact 7-billion-parameter open-weight model that used grouped-query attention and sliding-window attention to match or beat larger models, showing that architectural efficiency could substitute for raw parameter count.
Editorial record
Plain-language summary
Mistral 7B combines grouped-query attention (which shares key/value projections across query heads to cut memory and speed up inference) with sliding-window attention (which limits each token's attention to a fixed recent window so longer sequences stay cheap). With these changes it performed on par with or better than models roughly twice its size on common benchmarks. Released with downloadable weights under a permissive Apache 2.0 license, it became a practical base for local deployment and fine-tuning.
Knowledge graph
Relationships
Descendants
ExtendsEvidence: Strongly supported
LLaMA: Open and Efficient Foundation Language Models
Mistral builds on the open block
P-362
Source record
Provenance
- Record ID
- P-362
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2310.06825
- arXiv:2310.06825
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.