@hyunji_amy_lee: LLM agents & memory systems operate in continuously updated environments (Git repos, evolving docs). They must process …

LLM agents & memory systems operate in continuously updated environments (Git repos, evolving docs). They must process long contexts, recover earlier information, and reason over many updates that create interference between old and new information. How well do they handle this?

We introduce MINTEval: Frequent context changes & interference (avg. 86 updates) 5 challenging question types, including long-range lookback & reasoning over multiple targets distributed across context 4 realistic domains: state tracking, multi-turn dialogue, Wikipedia revisions, GitHub commits Avg. 138.8k tokens per instance (up to 1.8M) Human verification on generated QAs = 95.6%

Across 7 representative systems, MINTEval remains difficult, showing an avg. acc of 27.9%, and the best system reaches only 33.4%.

Our analysis shows: • Memory construction failures cause a 41.7% drop • Memory agents are highly sensitive to design choices • Memory systems have a strong bias toward insertion operations (76.8%) over deletion/update

Real-world information constantly evolves: codebases get updated, documents are revised, and user preferences change over time.

This naturally creates memory interference.

Under these interference-heavy settings, agents must:

• remember past states
• track updates over time
• reason across conflicting information
• resolve interference correctly

Most existing memory benchmarks fail to jointly capture: interference-heavy long-horizon contexts long-range lookback and multi-target aggregation questions evaluation across diverse domains

MINTEval addresses this gap with: heavy interference (avg. 86 updates) and long contexts (avg. 138.8k tokens, up to 1.8M) 5 question types, including challenging lookback and multi-target aggregation reasoning, and others, spread across 15.6k QA pairs 4 domains: state tracking (bAbI), multi-turn dialogue (Horizonbench), Wikipedia revisions, and GitHub commits

MINTEval evaluates 2 major capabilities / 5 question types under interference-heavy contexts:

Single-Target Recall: Can the agents retrieve the correct fact under heavy interference? Simple: retrieve the latest state (proactive interference) History: recover earlier states after many later updates (retroactive interference)

Multi-Target Aggregation: Can the agents combine information distributed across revisions? Ordering: recover temporal order of events Counting: aggregate occurrences across updates (e.g., how long a particular state persisted) Multihop: reason across multiple related targets (e.g., bridge / comparison questions)

We evaluate 7 representative systems including Full Context, RAG-based, and Memory-Augmented Agents on MINTEval, and they all struggle substantially. Average accuracy: 27.9% Best memory system (MemAgent): 33.4% Even SimpleMem + Gemini-3.1-Flash-Lite achieves only 30.3%, likely due to aggressive memory compression.

Performance drops sharply on: • History questions requiring long-range lookback (21.0%) • Multi-target aggregation questions that need to retrieve and reason over multiple pieces of info (26.5%) As compared to simple recall questions (47.5%).

We also observe limited cross-domain generalization: No single method consistently performs well across domains and question types.

Retrieval and memory construction remain the primary bottleneck.

• The largest performance drop comes from failures to retrieve or preserve the correct evidence: 41.7% degradation
• When the evidence is present, answering agents can still fail to use it correctly additional 25.2% drop

Longer lookback distances significantly hurt performance.

• As the distance between the queried fact and the current state increases, accuracy drops sharply, especially for Full-context methods and retrieval-based systems (RAG, HippoRAG)
• Memory-augmented agents also degrade as the distance increases, but show more robustness

Fewer memory update iterations improve performance.

• Using larger chunks (fewer memory modifications) generally leads to better performance; Frequent memory updates can introduce unintended overwrites or removals, making it harder to maintain coherent long-term memory.
• This especially hurts History and Counting questions, which require integrating information over long horizons.

These findings highlight the need for more robust memory management systems for interference-heavy, long-horizon environments across varying domains and questions.

https://arxiv.org/abs/2605.18565 https://huggingface.co/datasets/dinobby/MINTEval… https://github.com/amy-hyunji/MINTEval…

Awesome collaboration with @cyjustinchen (Co-Lead), @joykiratsingh, @codezakh, @EliasEskin, and @mohitban47 at @unc_ai_group @unccs @UTCompSci