Are We Ready For An Agent-Native Memory System?
Wei Zhou, Xuanhe Zhou, Shaokun Han, Hongming Xu, Guoliang Li, Zhiyu Li, Feiyu Xiong, Fan Wu
arXiv:2606.24775 · 124▲ · cs.CL, cs.DB, cs.IR
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
A benchmark study that decomposes 12 LLM agent memory systems (MemGPT/Letta, Mem0, Zep, A-MEM, MemOS, MemoryOS, Cognee, etc.) into four modules (representation/storage, extraction, retrieval/routing, and maintenance) and evaluates them from a data management angle. It runs end-to-end and ablation experiments across five workloads spanning 11 datasets, measuring not just task accuracy but retrieval fidelity, update robustness, long-horizon stability, and operational cost (index build time, query latency).
Why it matters
If you are choosing or building a memory layer for a long-running agent, this shows there is no single best architecture: the right choice depends on your workload bottleneck, and heavily structured graph systems can cost orders of magnitude more in index construction and query latency without proportional accuracy gains. It also flags concrete failure modes to watch: append-only stores return stale facts ('hallucinations of the past'), and semantic consolidation can destroy chronological cues needed for time-dependent queries.
Practical takeaway
When picking a memory system, match it to your workload: graph/temporal memory for cross-session fact aggregation, summary-first routing for exact recall in coherent dialogues, and trace-preserving (raw context) memory when operation order matters. Watch for retrieval accuracy dropping as the time gap between evidence and query grows, and prefer conservative/localized maintenance over global reorganization to control cost.
Key result
No single system dominated. On LongMemEval, Zep reached 48.0 LLM-Judge accuracy and Cognee 35.3 ROUGE-L F1; on LoCoMo exact match, MemOS led at 11.5 EM; on DB-Bench, plain Long Context scored best on EM (48.20) while MemoChat had the highest task success rate (55.40). For time-dependent queries, raw long-context retrieval still beat most dedicated memory systems, indicating consolidation often discards useful chronological information.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.