Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory

Hugging Face Daily Papers Papers
agent-memory benchmark implicit-association retrieval blind-spot

Summary

Introduces InMind Memory Bench, a benchmark to evaluate the implicit-association blind spot in agent memory systems, where retrieval-only memory fails to apply relevant world knowledge when surface similarity is low.

Long-term memory systems store what a user says in an external store and retrieve it when a related query arrives. This interface rests on an assumption so natural that it is rarely stated: a memory that is needed will resemble the query that needs it. World knowledge breaks the assumption. A tree-nut allergy should change the answer to a macaron request through their almond-flour ingredient, yet the two texts share no cue a retriever can see. We call this failure mode the implicit-association blind spot and introduce InMind, a 125-task, expert-verified benchmark spanning ten life domains, with 113 tasks grounded in citable public sources. Its paired controls separate three explanations that existing evaluations conflate: the fact was never stored, the model lacks the bridging knowledge, or the fact was stored and never surfaced. The verdict is clean. With the decisive memory placed in context, the backbone answers 84.0 percent of indirect queries; when the same memory must be retrieved, six vector, graph, and agentic memory systems reach at most 14.4 percent, even though they recall the same facts on demand at up to 100 percent. An embedding with eight times the dimensionality raises answer-blind target recall for every system yet leaves the gap essentially intact. A minimal diagnostic probe that keeps memory visible before the query arrives recovers most of the gap, locating the failure in the query-conditioned interface itself and pointing to routing, deciding which facts must stay visible, as the open problem InMind is built to score.
Original Article
View Cached Full Text

Cached at: 07/29/26, 03:50 AM

Paper page - Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory

Source: https://huggingface.co/papers/2607.24368 We want to make a claim:

Agent memory cannot be retrieve-only.

Retrieve-only memory is closer to a notebook than to human memory.

A notebook may store information perfectly, but it helps only when you realize that you should look something up. Human memory is different: important facts can continuously shape our judgment, and world knowledge connects things that appear unrelated on the surface.

Yet most agent-memory systems today are notebook-like. They write user information into an external store, then retrieve “relevant” memories when a new query arrives. Whether the implementation is a vector database, a knowledge graph, or agentic RAG, the basic order is usually the same:

First decide what is relevant from the query. Then let the model reason.

That order creates a structural blind spot.

A user says:

“I’m allergic to tree nuts.”

Later, they ask:

“I want to try making macarons. Any recipes?”

A good agent should warn them that traditional macarons use almond flour. But many memory systems simply provide a recipe: “tree-nut allergy” and “macarons” are not similar enough for the allergy memory to be retrieved in the first place.

Or consider this:

“I have lilies at home.” “I want to adopt a cat.”

A good agent should warn that lilies are highly toxic to cats and should be removed first. But a retrieve-only system may just give generic advice about caring for a new cat, because “lilies” and “cats” have almost no surface-level similarity.

The troubling part is that these systems may still answer direct questions perfectly:

“What am I allergic to?” “What plants do I have at home?”

So users are led to believe that the agent truly remembers—and will “keep in mind”—their important information.

But this can be afalse promise: the memory exists in the database, yet fails to affect the agent’s behavior when it actually matters.

We tested representative state-of-the-art memory systems, including HippoRAG 2, A-RAG, A-Mem, Mem0, MemoryOS, and xMemory. Across this class of cases, they show the same systematic failure pattern: they can recall the fact on demand, but fail to apply it when a world-knowledge bridge is needed.

To measure this phenomenon, we introduceInMind Memory Bench, a benchmark for theimplicit-association blind spotin agent memory.

Retrieval is necessary. But retrieval alone is not enough.

A reliable agent memory needs to answer a harder question: not onlywhat should be retrieved now?, but alsowhat information must remain available because it could matter in ways the current query does not reveal?

Similar Articles