MemoryLLM: Towards Self-Updatable Large Language Models

Editorial record

Plain-language summary

MemoryLLM embeds memory tokens as hidden vectors within each transformer layer, creating a large but fixed-size parametric memory pool. A self-update mechanism propagates new knowledge through all layers by extracting and updating memory slots while randomly dropping older slots, implementing exponential forgetting. The model demonstrates strong performance on model editing and long-context benchmarks while maintaining operational integrity through nearly a million memory updates without performance degradation.