Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

281reviewed records

22research domains

6 records

0012017Continual Learning & Memory

Peer reviewed

Overcoming Catastrophic Forgetting in Neural Networks

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Raia Hadsell, Demis Hassabis

The method estimates how sensitive prior-task performance is to each weight using the Fisher information, then adds a quadratic penalty anchoring the important weights while leaving the rest free to adapt. A single network could learn a sequence of tasks and retain earlier ones where ordinary training would overwrite them. It turned catastrophic forgetting from a vague failure into a measurable, tractable problem and anchored the continual-learning literature.

UnknownDifficulty 6/10Verified
Read record Primary source
0022025Continual Learning & Memory

Preprint

Titans: Learning to Memorize at Test Time

Ali Behrouz, Peilin Zhong, Vahab Mirrokni

Titans pairs short-term attention with a deep neural memory updated online by a surprise-based gradient signal, so salient new information is written into the memory as the model reads, with selective forgetting. This yields effective context well beyond the attention window while keeping inference cost bounded. It is a leading example of the test-time-learning branch, where memory lives in parameters updated at inference rather than only in the context window.

UnknownDifficulty 6/10Verified
Read record Primary source
0032025Continual Learning & Memory

Preprint

Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study

Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré

Cartridges train a smaller KV cache offline on a corpus by generating synthetic conversations and using context distillation, then load this trained cache at inference time. The Self-Study recipe produces Cartridges that match ICL performance on challenging long-context benchmarks while enabling 26.4x higher throughput. Cartridges extend effective context length (e.g., 128k to 484k tokens on MTOB) and can be composed at inference without retraining, amortizing training cost across all queries referencing the same corpus.

Fully open sourceDifficulty 7/10Verified
Read record Primary source
0042025Continual Learning & Memory

Preprint

Continual Learning via Sparse Memory Finetuning

Jessy Lin, Luke Zettlemoyer, Gargi Ghosh, Wen-Tau Yih, Aram Markosyan, Vincent-Pierre Berges, Barlas Oğuz

Sparse memory finetuning leverages memory layer models to update only the top-ranked memory slots based on TF-IDF scores computed against background pretraining data, reducing interference between new and existing knowledge. On fact learning and document QA tasks, this approach learns new knowledge to the same degree as full finetuning while showing substantially less forgetting—NaturalQuestions F1 drops only 11% with sparse memory finetuning versus 89% with full finetuning and 71% with LoRA.

UnknownDifficulty 6/10Verified
Read record Primary source
0052024Continual Learning & Memory

Preprint

MemoryLLM: Towards Self-Updatable Large Language Models

Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, Julian McAuley

MemoryLLM embeds memory tokens as hidden vectors within each transformer layer, creating a large but fixed-size parametric memory pool. A self-update mechanism propagates new knowledge through all layers by extracting and updating memory slots while randomly dropping older slots, implementing exponential forgetting. The model demonstrates strong performance on model editing and long-context benchmarks while maintaining operational integrity through nearly a million memory updates without performance degradation.

Fully open sourceDifficulty 6/10Verified
Read record Primary source
0062024Continual Learning & Memory

Preprint

Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning

Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, Meng Jiang

One PEFT Per User (OPPU) gives each user their own parameter-efficient fine-tuning module that stores user-specific behavior patterns and preferences parametrically, enabling model ownership and enhanced customization. By fine-tuning personal PEFT parameters on user history and integrating them with retrieval and profile augmentation, OPPU significantly outperforms existing prompt-based methods across seven LaMP benchmark tasks and demonstrates superior adaptation to user behavior shifts where history is less relevant.

Fully open sourceDifficulty 5/10Verified
Read record Primary source