IFCMemoryBench: Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval

# Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval
Source: [https://arxiv.org/html/2607.26072](https://arxiv.org/html/2607.26072)
\\@printpermissionfalse

\(2026\)

###### Abstract\.

Long\-term memory is becoming a core capability of LLM\-based agents, but existing evaluations largely test conversational recall in open\-domain or persona\-grounded settings\. We argue that a stronger test is whether an agent can reuse information from prior sessions while acting over a live, structured, domain\-specific environment\. We study this problem in Building Information Modelling \(BIM\), a professional engineering workflow where agents must query large IFC models while also relying on project specifications, client decisions, and engineering conventions that are often discussed in conversation but absent from the model\. We introduce IFCMemoryBench, a human\-validated benchmark for evaluating long\-term memory in LLM\-based BIM information retrieval\. IFCMemoryBench contains 143 multi\-session tasks across 19 projects and 4,016 prior sessions, derived from incomplete\-information questions in IFC\-Bench v2\. Each task seeds missing project context across earlier conversations and later asks a probe question that can be answered only by combining remembered context with live IFC queries\. Our evaluation framework decomposes memory performance into ingestion, retrieval, and utilization, and measures both answer quality and memory quality with expert\-validated LLM judges\. We evaluate representative vector\-, graph\-, and file\-based memory systems\. The strongest system achieves only 32\.4% answer accuracy under a deployment\-realistic ingestion scope, and remains below 60% under oracle\-filtered ingestion or a stronger probe agent\. Analysis shows that current general\-purpose memory systems often retrieve topically relevant context but store project knowledge as incomplete or fragmented facts\. These results reveal a domain\-transfer gap in agent memory and suggest that reliable professional agents require domain\-aware memory representations linking conversations, project knowledge, and structured model entities\.

Building Information Modelling \(BIM\), Industry Foundation Classes \(IFC\), Long\-term memory, LLM\-based agents, BIM information retrieval, Benchmark

††copyright:rightsretained††journalyear:2026††conference:KDD 2026 Workshop on Evaluation and Trustworthiness of Agentic AI; August 9, 2026; Jeju, Republic of Korea††ccs:Computing methodologies Natural language processing††ccs:Information systems Question answering††ccs:Computing methodologies Knowledge representation and reasoning††ccs:Applied computing Computer\-aided design## 1\.Introduction

As LLM\-based agents move from single\-turn assistants toward sustained, tool\-using systems, long\-term memory is becoming a core deployment capability\. Agents are increasingly expected to maintain project state, reuse information from prior interactions, and combine remembered context with live tool outputs\. However, most existing evaluations still treat memory largely as conversational recall: whether an agent can recover facts, preferences, or past events from earlier dialogue\. This setting is valuable, but it does not fully capture professional workflows, where remembered information must be used together with domain\-specific tools, structured data, and evolving project context\. In such settings, memory failures can lead not only to forgotten facts, but also to incomplete or misleading decisions\.

We study this problem in Building Information Modelling \(BIM\), a professional engineering workflow that provides a concrete stress test for long\-term memory in agentic AI\. BIM models are structured digital representations of building projects, containing typed objects such as walls, beams, spaces, systems, materials, quantities, and relationships\. Their dominant exchange format, the Industry Foundation Classes \(IFC\), is a large and semantically rich object\-oriented schema\. Querying IFC models requires domain expertise and technical skill, which has motivated recent work on LLM\-based BIM information retrieval, where agents answer natural\-language questions by inspecting IFC files through ReAct\-style tool use\(Gaoet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib3); Guoet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib4); Hellinet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib1)\)\.

Despite this progress, existing LLM\-based BIM retrieval systems and benchmarks remain largely stateless\. Benchmarks such as IFC\-Bench\(Hellinet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib1),[2025](https://arxiv.org/html/2607.26072#bib.bib17)\)evaluate independent question\-answer pairs, usually assuming that each answer can be derived from the IFC model alone\. Real BIM workflows are different\. Users return to the same project over weeks or months, refine requirements, confirm specifications, record client decisions, and correct earlier assumptions\. Much of this information never enters the IFC model, yet later questions may depend on it\. For example, a query may require IFC quantities together with an emission factor, cost assumption, room\-schedule correction, or engineering convention mentioned in a previous session\. A reliable BIM retrieval agent must therefore combine live IFC queries with long\-term project memory\.

This need reflects a broader gap in agent\-memory evaluation\. Existing memory systems use vector stores, knowledge graphs, or persistent files to retain information across sessions\(Zhonget al\.,[2024](https://arxiv.org/html/2607.26072#bib.bib5); Chhikaraet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib6); Rasmussenet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib7); Anthropic,[2025](https://arxiv.org/html/2607.26072#bib.bib13); LangChain,[2025](https://arxiv.org/html/2607.26072#bib.bib14)\), and benchmarks such as LongMemEval\(Wuet al\.,[2025a](https://arxiv.org/html/2607.26072#bib.bib8)\), MemoryAgentBench\(Huet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib9)\), and MemoryArena\(Heet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib10)\)evaluate important memory capabilities in open\-domain or persona\-grounded conversations\. However, they do not directly test whether memory remains reliable in a tool\-grounded professional workflow, where the agent must combine remembered context with structured, domain\-specific data\. In such settings, retrieving topically relevant memories is not enough: the memory must preserve exact project facts, units, entity references, assumptions, and their relationship to the structured model\.

To address this gap, we introduce IFCMemoryBench, a benchmark for evaluating long\-term memory in LLM\-based BIM information retrieval\. IFCMemoryBench converts incomplete\-information questions from IFC\-Bench v2 into multi\-session memory tasks\. Each task seeds missing project context across earlier chat sessions and later asks a probe question that can be answered only by combining remembered context with live IFC queries\. The benchmark contains 143 memory\-dependent tasks across 19 projects and 23 IFC models, with 4,016 prior chat sessions in total\.

Our evaluation decomposes memory into ingestion, retrieval, and utilization\. Ingestion measures whether prior sessions are written into memory in a useful form; retrieval measures whether the agent can access the needed remembered context; and utilization measures whether the agent can combine that context with IFC query results to produce the final answer\. We evaluate both answer quality and memory quality using expert\-validated LLM judges\.

We evaluate representative vector\-, graph\-, and file\-based memory systems\. The results reveal a substantial domain\-transfer gap\. Under a deployment\-realistic ingestion scope that does not assume prior knowledge of which messages contain durable facts, the strongest system achieves only 32\.4% answer accuracy\. Even with oracle\-filtered ingestion or a stronger probe agent, accuracy remains below 60%\. Current systems often retrieve context that is topically relevant but incomplete, fragmented, or weakly connected to IFC entities and project assumptions\. These findings suggest that reliable memory for professional agents requires domain\-aware representations linking conversations, project knowledge, and structured model entities\. This paper makes three contributions:

\(1\) We propose a methodology for converting stateless BIM information retrieval questions into multi\-session, memory\-dependent agent tasks\.

\(2\) We introduce IFCMemoryBench, a benchmark of 143 long\-term memory tasks across 19 projects and 4,016 prior sessions, together with an evaluation framework that measures both answer quality and memory quality\.

\(3\) We provide a systematic study of representative vector\-, graph\-, and file\-based memory systems for IFC\-grounded retrieval, exposing a domain\-transfer gap in current general\-purpose memory systems\.

## 2\.Background and Related Work

### 2\.1\.Memory in LLM\-based agents

The LLM\-agent literature borrows memory terms from cognitive science as engineering analogies for how long information persists and how it is reused\(Sumerset al\.,[2023](https://arxiv.org/html/2607.26072#bib.bib11); Wuet al\.,[2025b](https://arxiv.org/html/2607.26072#bib.bib12)\)\. Based on this taxonomy, we distinguish working, short\-term and long\-term memory\.Working memorydenotes the task context actively maintained while the agent handles a current user request, such as the instruction, intermediate state and recent tool outputs\.Short\-term memoryis the conversational history within one session: it keeps the agent coherent across a few turns, typically by appending past messages to the prompt or by summarising them\.Long\-term memorypersists across sessions and is itself usually subdivided intosemanticmemory \(decontextualised facts, constraints, preferences\),episodicmemory \(specific past interactions and events\), andproceduralmemory \(skills and routines\)\.

Our benchmark uses this distinction only as interpretive background\. Operationally, each task tests whether the agent can answer a BIM question by combining information read from the IFC model with project context seeded in prior chat sessions\. That context may contain semantic\-like project facts from schedules or specifications \(e\.g\. window glazing or room finish requirements\) as well as episodic\-like conversational updates \(e\.g\. a room\-schedule correction made in an earlier session\)\. Working and short\-term memory concern context management within the current task or session, whereas this study focuses on long\-term memory that persists across sessions; procedural skills are left to future work\.

### 2\.2\.LLM\-based BIM information retrieval

Early LLM\-based BIM information retrieval systems treated the task as natural\-language\-to\-query translation over IFC\-derived data, as in BIM\-GPT\(Zheng and Fischer,[2023](https://arxiv.org/html/2607.26072#bib.bib2)\)\. More recent work instead uses ReAct\-style agents that inspect IFC files through iterative tool calls, observing each result before choosing the next query or traversal\. Gao et al\.\(Gaoet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib3)\)introduce a schema\-guided multi\-agent framework that factors the task into specialised agents reasoning over the IFC schema and a shared tool layer\. Guo et al\.\(Guoet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib4)\)propose a multi\-agent alignment framework that maps user queries to BIM domain\-specific language and library\-code functions\. Hellin et al\.\(Hellinet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib1)\)generalise the trend toadaptive exploration, where the agent writes and executes Python code against the IFC, and release IFC\-Bench v2\.

A useful distinction across these systems is betweenclosed\-worldandopen\-worldsettings\. Most existing LLM\-based BIM information retrieval work is closed\-world: the question is fully answerable from the IFC itself, which cover direct property lookup and quantity calculations, etc\. Open\-world questions require external context not contained in the IFC, such as a project schedule, a structural specification, a designer’s intent\. Although such questions cannot be answered from the IFC alone, they are central to practical AEC workflows, where BIM queries often support design coordination, compliance checking, and client\-driven decisions\. In an agent\-assisted workflow\(Duet al\.,[2024](https://arxiv.org/html/2607.26072#bib.bib19); Denget al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib20); Duet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib21)\), this external context may be supplied by the user during previous conversations with the agent\. This suggests a role for long\-term agentic memory in retaining information that is not present in the IFC itself\. Our benchmark targets this open\-world regime\.

### 2\.3\.Agent memory systems and benchmarks

Long\-term memory systems persist selected information beyond the context window so it remains available in later sessions\. Practical implementations fall into three storage\-oriented designs\.Vector\-basedsystems store extracted facts, summaries, or interaction fragments as embeddings and retrieve them by semantic similarity, as in MemoryBank\(Zhonget al\.,[2024](https://arxiv.org/html/2607.26072#bib.bib5)\)and Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib6)\)\.Graph\-basedsystems represent memory as a knowledge graph that grows incrementally with the sessions and supports hybrid retrieval\(Rasmussenet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib7)\)\.File\-basedsystems keep memory as a plain text file that the agent reads for context and edits to add, update, or delete information, as in the MarkdownAGENT\.mdfiles of Claude Code\(Anthropic,[2025](https://arxiv.org/html/2607.26072#bib.bib13)\)and the filesystem\-backed memory in DeepAgents\(LangChain,[2025](https://arxiv.org/html/2607.26072#bib.bib14)\)\.

Long\-term memory has been studied with a parallel line of benchmarks\. LongMemEval\(Wuet al\.,[2025a](https://arxiv.org/html/2607.26072#bib.bib8)\)evaluates chat assistants on long\-term interactive memory; MemoryAgentBench\(Huet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib9)\)examines memory along four capabilities including accurate retrieval and conflict resolution; MemoryArena\(Heet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib10)\)focuses on interdependent multi\-session agentic tasks\. However, these general\-purpose memory benchmarks were not designed for our domain\.

### 2\.4\.Research gaps

Three gaps motivate this work\. First, BIM retrieval benchmarks such as IFC\-Bench treat tasks as independent, stateless queries and do not exercise long\-term memory\. Second, general memory benchmarks test conversational or textual recall, not the use of remembered project context alongside live tool\-based queries over structured IFC data\. Third, it remains unclear how different long\-term memory architectures perform on IFC\-grounded retrieval\. We address these gaps with an experimental framework for evaluating long\-term agentic memory in BIM information retrieval\.

## 3\.Methodology

### 3\.1\.Benchmark design

Each benchmark task asks an agent to answer a memory\-dependent project questionQtQ\_\{t\}at probe timett\. Before the probe, the agent has observed the prior chat sessionsS1,…,St−1S\_\{1\},\\dots,S\_\{t\-1\}, from which a memory systemffconstructs a memory store\. At probe time, the agent may also query the IFC model through query toolTIFCT\_\{\\mathrm\{IFC\}\}\. The produced answerAtA\_\{t\}is evaluated against a gold answerA^t\\hat\{A\}\_\{t\}and the project information seeded in the prior sessions\. The benchmark is agnostic to how memory is represented or accessed, requiring only that remembered project context can be used at probe time\. We model this process through three memory capabilities:

\(1\)Mi\\displaystyle M\_\{i\}=fingest​\(Mi−1,Si\),i=1,…,t−1,\\displaystyle=f\_\{\\mathrm\{ingest\}\}\(M\_\{i\-1\},\\,S\_\{i\}\),\\quad i=1,\\dots,t\-1,\(2\)Rt\\displaystyle R\_\{t\}=⋃j=1kfretrieve​\(qj,Mt−1\)⊆Mt−1,\\displaystyle=\\bigcup\_\{j=1\}^\{k\}f\_\{\\mathrm\{retrieve\}\}\(q\_\{j\},\\,M\_\{t\-1\}\)\\;\\subseteq\\;M\_\{t\-1\},\(3\)At\\displaystyle A\_\{t\}=LLM​\(Qt,Rt,TIFC\)\.\\displaystyle=\\mathrm\{LLM\}\(Q\_\{t\},\\,R\_\{t\},\\,T\_\{\\mathrm\{IFC\}\}\)\.whereMt−1M\_\{t\-1\}is the memory store accumulated from the prior sessions, andRtR\_\{t\}is the part of that store retrieved for the probe\. \(1\)Ingestion\(Eq\.[1](https://arxiv.org/html/2607.26072#S3.E1)\) integrates the prior sessions into memory*sequentially*, merging each sessionSiS\_\{i\}into the current storeMi−1M\_\{i\-1\}to preserve useful information while filtering distractor questions, routine tool traces, and conversational noise\. This write step is*online*and*stateful*: each sessionSiS\_\{i\}is compared with the accumulated storeMi−1M\_\{i\-1\}, and the system decides what to keep, summarise, overwrite, or discard\. This is what separates long\-term memory from retrieval\-augmented generation \(RAG\): RAG indexing is lossless and append\-only, storing content without changing previous entries and leaving selection to retrieval at query time\. In contrast, memory makes selective decisions during writing and integrates new content into a store that evolves across sessions\. \(2\)Retrieval\(Eq\.[2](https://arxiv.org/html/2607.26072#S3.E2)\) is the LLM agent dynamically querying a memory retrieval function, and we distinguish two sides\.*System\-side*retrieval is the query functionfretrievef\_\{\\mathrm\{retrieve\}\}provided by the memory system: given a queryqjq\_\{j\}, it returns a fixed subset of the storeMt−1M\_\{t\-1\}\.*Agent\-side*retrieval is how the probe LLM agent uses this function: the agent chooses how many querieskkto issue and writes eachqjq\_\{j\}itself, soRtR\_\{t\}is the union over thekkresults\. Which part ofMt−1M\_\{t\-1\}this covers therefore depends on what the agent decides to query\. \(3\)Utilization\(Eq\.[3](https://arxiv.org/html/2607.26072#S3.E3)\) combines the retrieved memoryRtR\_\{t\}, the probe questionQtQ\_\{t\}, and IFC results fromTIFCT\_\{\\mathrm\{IFC\}\}to produceAtA\_\{t\}, while keeping remembered context distinct from information read directly from the IFC model\. This separation lets us later diagnose failures according to whether they arise in memory writing, memory retrieval, or answer\-time use\.

### 3\.2\.Dataset construction

The dataset is built on IFC\-Bench v2\(Hellinet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib1)\), which groups questions by increasing information demand\. Categories 1–3 are closed\-world tasks answerable from the IFC model itself, whereas Category 4 covers incomplete\-information cases in which the required facts are absent, ambiguous, or must be estimated from sources outside the model\. We treat these Category 4 questions as seeds for multi\-session memory tasks: the missing external information is supplied through earlier chat sessions, and the original question is asked only afterwards as the probe\. The remaining steps, summarised in Figure[1](https://arxiv.org/html/2607.26072#S3.F1), describe how each task is built\.

Extracting the IFC\-side information\.We run three different high\-performing information\-retrieval systems on all Category 4 questions from IFC\-Bench v2\. Each run produces, for every question, a trace of the IFC information retrieved while answering it\. An LLM then cross\-analyses the three traces for each question to derive the union of IFC information the question requires: which entities, property sets, quantities, and spatial containments mattered\. The result is a precise account of what the IFC supplies\.

Filtering by answerability\.A second LLM compares the extracted IFC information against the IFC\-Bench v2 ground\-truth answer and assigns one of three memory\-dependent labels:not\_in\_ifc\(the IFC contains essentially nothing relevant to the question\),needs\_external\_info\(the IFC supplies context, but the answer hinges on information beyond the model\), andpartially\_answerable\_from\_ifc\(the IFC yields part of the answer but leaves gaps or ambiguities\)\. All three labels signal a dependence on remembered context and are kept; any question that proves fully answerable from the IFC alone is discarded, since it would not exercise memory\.

Synthesising missing context and the gold answer\.For each kept task, an LLM agent equipped with web search and a code interpreter is given the assigned label, the extracted IFC information, and the original IFC\-Bench v2 question, and produces two outputs\. The first is theextra informationa perfect answer needs but the IFC lacks: the agent retrieves external general facts such as standards, EPD/GWP emission factors and unit costs through web search, and uses the code interpreter for aggregation or unit conversion\. Missing project\-specific documents that cannot be looked up, such as schedules or specifications, are instead synthesised as plausible project context consistent with the IFC information\. The second output is the correspondinggold answer, which combines the extra information with the IFC context and serves as the ground\-truth answer for the probe\. Figure[1](https://arxiv.org/html/2607.26072#S3.F1)illustrates this step with the embodied\-carbon task\. Theextra informationbecomes the hidden target memory content\.

Generating realistic prior chat sessions\.A dataset\-generation agent then synthesises between2525and4040prior chat sessions per task\. Each session contains a small set of user messages that combine realistic BIM information retrieval queries \(about levels, walls, materials and schedules\), natural distractors, and natural reintroductions of the project knowledge derived from theextra information\. The seeded information appears as specifications mentioned in passing, user corrections to the agent, or background context for unrelated requests, so that it is distributed across sessions, wordings and surrounding dialogue\. To convert these messages into realistic human\-agent conversation, we replay each synthetic session against the same IFC model using a ReAct agent equipped with an IFC query CLI tool\. The agent answers turn by turn\. At each turn, it can access only the current user message, previous dialogue within the same chat session, the IFC model through the CLI tool, and its own tool outputs\. This restriction prevents facts from other sessions, the hiddenextra information, or the gold answer from leaking into the generated dialogue\. We store the complete interaction, including user messages, assistant messages, tool calls and tool outputs, as the session’sturns\. Finally, the original Category 4 question is paraphrased into a finalprobe questionthat does not reveal the gold answer but explicitly invites the agent to consult both memory and the IFC\.

Every step of the dataset construction pipeline is validated by human review, which discards any case where the gold answer is incorrect, where the seeded information conflicts with the IFC, or where the probe is ambiguous\. The resulting benchmark contains 143 tasks\. Each task pairs a probe question with its gold answer and a set of prior chat sessions where user messages scatter the project knowledge needed to answer it, for a total of 4 016 prior sessions across the benchmark\.

![Refer to caption](https://arxiv.org/html/2607.26072v1/x1.png)Figure 1\.IFCMemoryBench construction pipeline\. An IFC\-Bench v2 Category 4 question is turned into a memory\-dependent task, with one embodied\-carbon example threaded through all stages\.
### 3\.3\.Evaluation metrics

Each system is evaluated along two independent axes:answer qualityandmemory quality\. Each task is evaluated with an LLM\-as\-judge protocol using the probe, gold answer, system response, the project information seeded in prior sessions, and the memory retrieved during the run\.

Answer judge\.The answer judge assesses the final response along three binary dimensions: \(1\)correctness, whether its facts and conclusions match the gold answer, allowing for paraphrase and small rounding differences; \(2\)completeness, whether it provides the central result and any qualifiers needed to avoid misleading the user; and \(3\)relevance, whether it addresses the probe rather than a neighbouring question, project, or element\. A task is counted as answer\-correct only when all three dimensions are true, and we report*answer accuracy*as the percentage of answer\-correct tasks\. This design follows existing evaluation practice in BIM question answering and agent\-memory benchmarks\(Hellinet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib1); Huet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib9)\), and reduces sensitivity to verbosity differences across memory systems\.

Memory judge\.The memory judge assesses how well the project information seeded in prior sessions was preserved in memory, retrieved at probe time, and used in the final response\. It scores three binary dimensions: \(1\)retrieval\_relevant, whether the retrieved memory is about the right probe, project and elements; \(2\)retrieval\_covers\_key\_facts, whether the retrieved memory preserves enough of the seeded project information to supply the hidden facts needed for the answer; and \(3\)answer\_uses\_memory, whether the final answer uses the retrieved memory rather than fabricating or relying only on IFC information\. A task is counted as memory\-correct only when all three dimensions are true, and we report*memory accuracy*as the percentage of memory\-correct tasks\.

Human validation of the judges\.To confirm that both judges track expert judgement, one domain expert independently re\-labelled a random sample of 40 tasks drawn across the system runs of the main controlled comparison\. On this single sample the expert assigned, for each task, both an answer\-correctness and a memory\-correctness label under the rubrics of Sec\.[3\.3](https://arxiv.org/html/2607.26072#S3.SS3)\. For answer\-correctness the expert and the judge agreed on 38 of the 40 tasks \(95%95\\%raw agreement, Cohen’sκ=0\.90\\kappa=0\.90\); for memory\-correctness they agreed on all 40 tasks\. Both values lie in the almost\-perfect band of the Landis–Koch scale\. The two answer\-judge disagreements were boundary cases under the correctness rubric rather than substantive errors: in one, the system answer differed from the gold value only by a rounding/unit\-precision margin that the expert accepted but the judge did not; in the other, a qualifier the expert considered necessary for completeness was present only implicitly in the response\. Such boundary cases are specific to graded answer correctness; the memory dimensions instead turn on whether the retrieved context is on\-topic, covers the seeded facts, and is actually used, which the expert and the judge read off consistently, leaving no disagreements on this sample\. Neither answer\-judge disagreement altered the relative ranking of the affected systems, so we use the judge labels for all results reported below\.

### 3\.4\.Studied memory systems

All tested systems use the same ReAct\-style BIM retrieval agent: the agent receives the probe, a fixed prompt policy, and read\-only access to IFCQuery, a CLI for querying IFC model information\(IfcOpenShell,[2025](https://arxiv.org/html/2607.26072#bib.bib15)\)\. We vary only the attached memory layer: how prior sessions are ingested, how relevant context is retrieved, and how that context is exposed to the agent\. A no\-memory condition discards prior sessions and serves as the lower bound/baseline\.

Vector memory\.The vector\-memory condition instantiates the Mem0 pattern\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib6)\)\. During ingestion, prior conversation messages are sequentially passed through a memory\-extraction pipeline that distils them into standalone textual memory items\. These items are embedded and stored in an item\-scoped vector database collection, together with metadata that separates one benchmark task from another\. During retrieval, the agent’s query is embedded and used for top\-kksemantic similarity search over that vector collection\. The only memory surface exposed to the probe agent is asearch\_project\_memory\(query, limit\)tool that returns the ranked memories as text\.

Temporal graph memory\.The graph\-memory condition instantiates the Graphiti temporal\-knowledge\-graph pattern\(Rasmussenet al\.,[2025](https://arxiv.org/html/2607.26072#bib.bib7)\)\. During ingestion, each prior session is stored as a time\-stamped episode, which acts as the source record for facts extracted from that conversation\. Graphiti then creates nodes for important entities mentioned in the session and edges representing factual relationships between them, with temporal validity fields where available\. Retrieval uses hybrid search that combines semantic and keyword matching over the graph, then surfaces structured results such as relation facts, entity summaries and validity timestamps\. To keep the agent interface comparable with the vector condition, these results are again exposed throughsearch\_project\_memory\(query, limit\), but the returned text is generated from graph records rather than from a flat vector index\.

File\-based memory\.The file\-memory condition instantiates an agent\-managed Markdown memory pattern using DeepAgents\(LangChain,[2025](https://arxiv.org/html/2607.26072#bib.bib14)\)\. During the ingestion, a dedicated memory\-writing agent processes the prior sessions one at a time, deciding after each session whether to write or update a compact project memory file \(memory\.md\)\. We test two file\-memory forms\. The uncited form exposes only the Markdown summary\. The cited form writes the same kind of Markdown summary but adds stable message citations such as\[S003\-M02\], which point into a companionprior\_sessions\.jsonarchive of the original messages for optionalgrep/read\_fileverification\. At probe time,memory\.mdis injected directly into the probe agent’s system prompt as read\-only project context, rather than retrieved through a search tool\.

## 4\.Experiments and Results

### 4\.1\.Benchmark statistics

Table[1](https://arxiv.org/html/2607.26072#S4.T1)summarises the final benchmark\. The 143 tasks cover 19 projects and 23 IFC models spanning architectural, structural, MEP, plumbing, ventilation and city disciplines\. The benchmark contains in total 9 488 user messages, 45 983 assistant messages and 39 532 tool messages\. All project information seeded into each task are synthetic or derived from publicly available sources\. The IFC models are drawn from the public IFC\-Bench\(Hellinet al\.,[2026](https://arxiv.org/html/2607.26072#bib.bib1),[2025](https://arxiv.org/html/2607.26072#bib.bib17)\)\.

Table 1\.IFCMemoryBench dataset statistics\.
### 4\.2\.Experimental setup

All main experiments use the same ReAct probe agent implemented with the DeepAgents framework\. The agent is equipped with a bash execution tool for runningifcquery; unrelated DeepAgents tools are removed so that IFC access is mediated through this interface\. To keep the comparison fair, Grok\-4\.3 is used for the probe agent, the LLM judges, and the LLM\-based components inside the tested memory systems, with temperature 0\. We instantiate each memory system with its standard default settings except where noted\. Mem0 stores its memory items in a Qdrant vector database with the default embedding model\. Graphiti is backed by Neo4j; we keep its default embedder, graph schema, and hybrid retrieval, which fuses lexical \(BM25\) and dense semantic candidates by reciprocal\-rank fusion \(RRF\)\. The file memory is built on the DeepAgentsMemoryMiddleware, which reloadsmemory\.mdfrom disk at each step\.

### 4\.3\.Memory system comparison

The decomposition in Sec\.[3\.1](https://arxiv.org/html/2607.26072#S3.SS1)also sets the control logic for the experiments\. In this main comparison, we fix the probe agent and vary the memory layer, so observed differences are attributed to how each memory architecture stores and exposes project context to the same agent\. Section[4\.7](https://arxiv.org/html/2607.26072#S4.SS7)uses the complementary control: it fixes the stored memory and varies the probe LLM to examine how agent capability affects agent\-side retrieval and memory use\.

This main comparison fixes the ingestion scope to user messages plus the assistant’s final answer, rather than the user\-messages\-only scope that scores highest in Sec\.[4\.4](https://arxiv.org/html/2607.26072#S4.SS4)\. This reflects deployment realism: a memory layer running over a live conversation cannot know in advance which turns carry durable project facts, whereas the user\-only scope presupposes a perfect pre\-filter and acts as a near\-oracle ingestion scope rather than an operating point a deployed system could assume \(not the full\-context oracle of Sec\.[4\.5](https://arxiv.org/html/2607.26072#S4.SS5), which bypasses the memory layer entirely\)\. Reporting the deployment\-faithful scope as the headline gives a conservative estimate; Sec\.[4\.4](https://arxiv.org/html/2607.26072#S4.SS4)then isolates how much of the gap is attributable to ingestion scope alone\.

Table[2](https://arxiv.org/html/2607.26072#S4.T2)reports answer and memory\-judge scores\. We analyse the results from three perspectives: how prior sessions are ingested into memory, how useful memory is retrieved or surfaced at probe time, and how the agent uses that memory in the final answer\. The no\-memory lower bound answers no task fully, confirming that the benchmark requires cross\-session memory; yet even the best system under this fixed\-probe setting reaches only 32\.4% answer accuracy\. The Wilson 95% intervals around these accuracies overlap for several systems, so we do not over\-interpret small gaps in the ranking; what the intervals make unambiguous is that every system performs poorly in absolute terms, with all upper bounds far below the level a reliable assistant would need\. We therefore ask where failures become visible across memory ingestion, retrieval, and utilization in the final answer\.

Table 2\.Per\-dimension results on IFCMemoryBench\. Each value is the percentage of tasks for which the corresponding binary \(yes/no\) judge returns positive\. The answer judge scores correctness \(Corr\.\), completeness \(Cmp\.\) and relevance \(Rel\.\); answer accuracy \(Ans\-acc\.\) requires all three to hold\. The memory judge scores whether the retrieved context is relevant \(R\-rel\.\), covers the key facts \(R\-cov\.\) and is actually used in the answer \(A\-mem\.\); memory accuracy \(Mem\-acc\.\) requires all three to hold\. Bracketed values under each rate are Wilson score 95% confidence intervals for the underlying binomial proportion over the evaluated tasks\(Wilson,[1927](https://arxiv.org/html/2607.26072#bib.bib18)\)\.The first comparison concerns what is written from prior sessions\. Under the fixed Grok probe agent, Mem0 performs best overall, but the two Markdown variants isolate the effect of written memory quality because they share the same storage and injection mechanism\. The cited variant has higher observed scores than the uncited one, suggesting that memory should preserve traceable, self\-contained project facts rather than compressed session summaries\. Graphiti shows the same issue from another angle: relevant graph records do not necessarily expose facts in a form the agent can reuse\.

The main bottleneck is access coverage\. Across mechanisms, the agent usually receives context from the right project and topic, but that context often omits key facts\. This relevance–coverage gap is clearest for Graphiti, where specification\-level facts may be split across nodes and edges instead of returned as coherent evidence\. Mem0 and cited Markdown reduce this problem, but even the best system covers only about half of the required facts\.

Finally, under this fixed agent,A\-mem\.indicates that useful memory is often used when it is available\. Since these scores exceed coverage scores, the immediate bottleneck in the main comparison is often missing facts in the accessed context rather than wholesale failure to use memory\. This also explains why answer relevance is high while completeness remains much lower\. Overall, the remaining failures mainly reflect weaknesses in memory representation and access, though Sec\.[4\.7](https://arxiv.org/html/2607.26072#S4.SS7)shows that agent\-side search effort can also materially change coverage and answer quality\.

### 4\.4\.Effect of ingestion scope

Reliable cross\-session reuse depends not only on the memory architecture but also on what is written to memory from each prior session\. We therefore hold storage, retrieval and probe fixed, and vary only the ingestion scope: only user messages, user messages plus the assistant’s final answer, and full turns including intermediate reasoning, tool calls and outputs\. Across all three systems, the best results come from ingesting only user messages \(Table[3](https://arxiv.org/html/2607.26072#S4.T3)\)\. This finding is partly a consequence of the benchmark design, and should therefore be read with that limitation in mind\. In our construction \(Sec\.[3\.2](https://arxiv.org/html/2607.26072#S3.SS2)\), durable facts are seeded only in user messages; assistant and tool turns do not introduce additional durable facts\. Wider scopes therefore add mostly distractors\. As this noise increases, Mem0 is the most stable, Graphiti degrades when assistant and tool text is converted into extra graph nodes, and the Markdown file memory is least robust\.

The cost effect is also substantial: per task, ingestion grows from 0\.9 k tokens with user messages to 10\.2 k with final answers and 167 k with full turns \(180x\)\. Although ingestion time usually follows this increase, successful full\-turn runs average only about 215 s because the agent often writes no memory file under heavy noise\. Full turns thus pair the highest read cost with ingestion collapse, making scope selection a central design choice rather than neutral preprocessing\.

Table 3\.Effect of ingestion scope\. Quality is answer accuracy and memory accuracy \(as in Table[2](https://arxiv.org/html/2607.26072#S4.T2)\); ingestion cost is the tokens read per task and the mean per\-task ingestion time\. Full turns was run only for the file memory due to budget constraints\.
### 4\.5\.Full\-context oracle as a costed upper bound

We additionally evaluate full\-context oracle conditions in which the probe agent receives prior messages directly in its prompt, without memory extraction or retrieval\. These conditions are not intended as deployable memory baselines: they assume access to the task\-relevant prior sessions and, in the cleanest setting, to the message type that contains the seeded durable facts\. Instead, they serve as sanity\-check upper bounds on task answerability and make the cost model of long\-context prompting explicit\.

When all prior user messages are exposed losslessly, memory accuracy reaches 95\.1% and answer accuracy reaches 83\.2%, showing that most benchmark tasks are solvable when the relevant project facts are directly visible to the agent\. However, as shown in Figure[2](https://arxiv.org/html/2607.26072#S4.F2), this accuracy is obtained by paying the history cost at probe time for every query\. In contrast, memory systems pay an up\-front ingestion cost and expose a much smaller context at probe time, allowing this write cost to be amortized across future project queries\.

The oracle also shows that more context is not automatically better\. When we expand the prompt from prior user messages to user messages plus assistant final answers, memory accuracy drops from 95\.1% to 74\.1%, and answer accuracy drops from 83\.2% to 44\.1%\. This degradation indicates that naive long\-context prompting is sensitive to conversational noise: additional assistant\-side text can obscure, dilute, or conflict with durable project facts\. The gap between the clean user\-message oracle and the tested memory systems therefore quantifies the loss introduced by current memory\-writing, representation, and access mechanisms, rather than an inherent ambiguity in the benchmark\.

![Refer to caption](https://arxiv.org/html/2607.26072v1/x2.png)Figure 2\.Cost–quality trade\-off between full\-context prompting and memory systems\. The x\-axis shows probe\-time input tokens per task, the y\-axis shows answer accuracy, and bubble area denotes one\-time ingestion tokens per task\. Full\-context prompting gives a costed upper bound, while memory systems amortize ingestion across future queries but currently lose substantial quality during memory writing and access\.
### 4\.6\.Effect of domain customisation

The experiments so far use each memory system’s default, general\-purpose ingestion policy\. We ask whether a light domain customisation, telling the memory layer what an AEC/IFC memory should contain, narrows the coverage gap without changing the architecture or retrieval\. Holding the agent, judges, model, retrieval and ingestion scope \(user\+\+final answer\) fixed, we apply one minimal customisation per system: for Mem0, a domain instruction in its fact\-extraction prompt that keeps exact quantities, units and element or material names as self\-contained facts; for the DeepAgents file memory, a domain template that separates IFC\-read facts from user\-supplied facts and records each external value atomically under a fixed schema\. We customise only these two systems, as Graphiti’s equivalent adaptation requires redesigning its entity and edge ontology rather than a prompt, which we leave to future work\.

The two systems respond in opposite directions \(Figure[3](https://arxiv.org/html/2607.26072#S4.F3)\)\. In our experiments, the domain template was the more effective of the two memory\-layer changes for the agent\-managed file memory: it raised both answer and memory accuracy, bringing the file memory roughly level with the default Mem0 store, apparently by making the retrieved context more relevant and more complete\. The same instruction does not help Mem0 and slightly lowers both scores, since its pipeline already distils standalone facts and the extra rule mainly fragments each specification, reducing per\-item coverage\. The value of customisation thus depends on the default ingestion quality: a weak, generic writing policy benefits clearly, whereas an already strong one gains little and can regress\.

![Refer to caption](https://arxiv.org/html/2607.26072v1/x3.png)Figure 3\.Effect of domain customisation\.
### 4\.7\.Effect of LLMs used in the probe agent

The main study fixes the probe agent, the judges and the memory\-system LLMs to Grok\-4\.3 for a fair comparison across architectures \(Sec\.[4](https://arxiv.org/html/2607.26072#S4)\); here we change only the probe agent’s LLM to Gemini\-3\.5\-Flash and hold everything else fixed, reusing the previously ingested memory verbatim\. Any change in scores therefore reflects the effect of the probe LLM on agent\-side retrieval and memory*utilization*, not on how the memory was written\.

Each of the 143 tasks is answered by both agents from the same fixed memory, so the two probe LLMs are compared on identical items as 143*paired*correct/incorrect outcomes\. We test a single quantity: the accuracy gainΔ=pGemini−pGrok\\Delta=p\_\{\\mathrm\{Gemini\}\}\-p\_\{\\mathrm\{Grok\}\}between the two LLMs on these items, whose posterior we estimate with a Bayesian paired test of proportions\(Vosseler,[2026](https://arxiv.org/html/2607.26072#bib.bib16)\)\(exact Pólya–Gamma sampler, so no large\-sample normal approximation is needed atn=143n\{=\}143\)\. From the posterior we report the meanΔ\\Delta, its 95% credible interval \(CrI\) andP​\(Gemini\>Grok\)P\(\\mathrm\{Gemini\}\>\\mathrm\{Grok\}\)\. We call a gain*decisive*when its 95% CrI excludes0\(its plausible range lies entirely on one side of0, so the sign of the gain is settled\), and*undecided*when the CrI still includes0\(the data do not yet resolve whether the gain is real\)\. We use a*paired*test because both agents see the same tasks, so pairing cancels task\-difficulty noise, and a Bayesian one because the posterior gives a directly interpretable probability and effect size rather than a singlepp\-value\.

Gemini improves both answer and memory accuracy on all three systems \(Figure[4](https://arxiv.org/html/2607.26072#S4.F4)\(b\)\): decisively for the file memory and Mem0, and positively but undecidedly for Graphiti, whose 95% CrI still includes0\. The gain even reorders the systems: the file memory leads under Gemini, whereas the Mem0 store leads under Grok \(Table[2](https://arxiv.org/html/2607.26072#S4.T2)\), so the ranking of memory architectures depends on the agent, not on the memory layer alone\.

The cause is search effort: Gemini issues several times more memory queries than Grok \(e\.g\. 8\.1 vs\. 1\.7 per task for Mem0\) and surfaces more context, at a several\-fold latency cost\. As Figure[4](https://arxiv.org/html/2607.26072#S4.F4)\(a\) shows, this mainly raises the completeness and retrieval\-coverage dimensions, while relevance is the only dimension that falls; since each overall accuracy \(the bold rows\) is the conjunction of its three dimensions, it rises only as far as its weakest dimension allows\. The file memory gains most, as its content is injected in full and its citations can be verified on demand, so all of its dimensions improve together; the graph gains least, its facts staying fragmented so that retrieval\-coverage remains low even for Gemini\. A more capable agent thus narrows but does not close the coverage gap, confirming that utilization, not only ingestion, is a primary lever, and that the probe LLM should be reported as a first\-class experimental factor\.

![Refer to caption](https://arxiv.org/html/2607.26072v1/x4.png)Figure 4\.Effect of the probe\-agent LLM, with memory held fixed\. \(a\) Overall answer and memory accuracy and the judge dimensions that compose them\. \(b\) Posterior accuracy gainΔ\\Delta\(Gemini−\-Grok\), one row per system and metric\. The dot is the posterior mean, the bar its 95% credible interval \(CrI\), and the dashed line marksΔ=0\\Delta\{=\}0\(no difference\)\.

## 5\.Discussion

IFCMemoryBench is a synthetic, human\-validated benchmark whose design rests on a few assumptions\. First, task\-relevant prior knowledge is seeded only through user chat messages \(Sec\.[3\.2](https://arxiv.org/html/2607.26072#S3.SS2)\), while real projects may also rely on uploaded files, drawings or other external artefacts; extending the benchmark to these sources is an important direction for future work\. Second, as Sec\.[4\.4](https://arxiv.org/html/2607.26072#S4.SS4)notes, user\-only ingestion scoring best is partly an artefact of seeding durable facts only in user turns\. This limits ecological validity: in real projects, assistant replies and tool outputs can themselves carry memorable content, such as computed quantities, intermediate results, or established query routines, that a deployed memory system would need to retain\. A future version of the benchmark will therefore seed durable facts into assistant and tool turns as well, turning ingestion\-scope selection into a genuine selective\-memory problem rather than a near\-trivial “keep only user messages” rule, and enabling evaluation of procedural memory such as reusable IFC query routines\. Third, a single model family \(Grok\-4\.3\) simultaneously serves as the probe agent, the LLM components inside the memory systems, and the answer and memory judges, which in principle exposes the study to self\-preference bias, the tendency of an LLM judge to favour outputs from its own family\. In the main comparison this exposure is uniform across systems, since every answer is produced by the same agent, so it does not distort the relative ranking of memory layers; it is most relevant to the probe\-LLM comparison \(Sec\.[4\.7](https://arxiv.org/html/2607.26072#S4.SS7)\), where a Grok judge scores both Grok and Gemini agents\. Two design choices bound the risk: every judgement is anchored to a human\-authored gold answer and a fixed binary rubric rather than to open\-ended preference, and the judges were validated against a blind expert relabelling with almost\-perfect agreement \(Sec\.[3\.3](https://arxiv.org/html/2607.26072#S3.SS3)\)\. Cross\-model judging remains a valuable robustness check for future work\. We thus view IFCMemoryBench as, to our knowledge, the first domain\-specific long\-term\-memory benchmark for BIM information retrieval, and as an extensible basis for broader sources and memory types\.

A further question is whether a dedicated memory layer is needed at all, since long\-context models ingest hundreds of thousands of tokens and the prior sessions per task here are small enough to concatenate directly into the prompt\. Our full\-context oracle makes this concrete: exposing all prior user messages losslessly reaches 83\.2% answer accuracy \(Sec\.[4\.5](https://arxiv.org/html/2607.26072#S4.SS5), Fig\.[2](https://arxiv.org/html/2607.26072#S4.F2)\)\. We read this as a property of the present task instances, not evidence against memory systems\. First, the benchmark scales beyond any fixed context window: adding prior sessions and synthetic distractors degrades the relevant\-to\-irrelevant ratio arbitrarily while the answer still turns on a few seeded facts, and our full\-turn ingestion result already shows indiscriminate ingestion collapsing at the upper end \(about 167k tokens per task, Sec\.[4\.4](https://arxiv.org/html/2607.26072#S4.SS4)\)\. Second, production deployments accumulate effectively unbounded history across months and many concurrent users, which cannot be reloaded in full on every query for cost and latency reasons: the oracle pays this probe\-time cost on every query, whereas memory systems amortize it through up\-front ingestion\. Selective storage and retrieval are therefore required, and the bottleneck the benchmark exposes, surfacing the right project facts, is exactly what a long\-context dump does not resolve\. We thus treat the full\-context oracle as a costed upper bound and sanity check, not a substitute for the memory capability under study\.

Within this scoped setting, the experiments surface a set of interacting challenges that span memory ingestion, access, and utilization\. We frame these as potential limitations and areas for improvement that the benchmark makes visible, rather than as inherent failures of the systems studied\. The main experiments suggest that ingestion and representation are the primary upstream limitation: project knowledge may be written incompletely, inverted into an “absent from the model” finding, or fragmented into isolated facts that are harder to use for list, comparison or aggregation probes\. These writing limitations in turn appear to constrain retrieval coverage: agents often access topically relevant memory, but not always the complete project facts needed for the answer\. Section[4\.7](https://arxiv.org/html/2607.26072#S4.SS7)adds that, with stored memory held fixed, the probe LLM still materially affects agent\-side retrieval and memory use through its search effort and evidence verification\. Taken together, these observations suggest that the challenge is less about retrieving more text and more about preserving project knowledge in a structured, traceable form and pairing it with agents capable of using that memory alongside the live IFC\.

We regard surfacing such limitations as a core value of the study rather than a negative result\. Because IFCMemoryBench is a controlled research instrument applied to research prototypes, it can isolate where current general\-purpose memory systems lose project knowledge in a low\-risk setting, before such behaviour would matter in practice\. Making these areas for improvement explicit and measurable is what allows the community to target them, and we hope the benchmark can serve as a shared diagnostic for tracking progress on domain\-grounded memory\.

## 6\.Conclusions

This paper introduced IFCMemoryBench, a benchmark and extensible methodology for synthesising multi\-session memory tasks from existing BIM question\-answering data, and used it for an exploratory study of representative general\-purpose memory systems in IFC\-grounded BIM information retrieval, intended to highlight areas for improvement of existing systems\. Our observations point to a domain\-transfer opportunity: such systems can often retrieve related context, but have more room to preserve project knowledge as complete, grounded evidence usable together with live IFC queries, suggesting that BIM memory may benefit from being organised around the structure of project information rather than treated as a generic conversation\-memory layer\. The probe\-LLM comparison further shows these rankings are conditioned on the agent, making the LLM itself an important experimental factor\. We offer domain\-adapted memory designs that connect conversations, project documents and IFC entities, so that remembered knowledge could better support traceable, measurement\-based reasoning, as an open research direction\.

###### Acknowledgements\.

The authors would like to thank Sylvain Hellin and Stefan Fuchs for their valuable suggestions and inspiring discussions\.

## References

- Anthropic \(2025\)Claude code\.Note:Computer software[https://www\.anthropic\.com/claude\-code](https://www.anthropic.com/claude-code)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p1.1)\.
- P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav \(2025\)Mem0: building production\-ready AI agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.19413)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2607.26072#S3.SS4.p2.1)\.
- Z\. Deng, C\. Du, S\. Nousias, and A\. Borrmann \(2025\)BIMgent: towards autonomous building modeling via computer\-use agents\.InWorkshop on Computer\-Use Agents at ICML,External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.07217)Cited by:[§2\.2](https://arxiv.org/html/2607.26072#S2.SS2.p2.1)\.
- C\. Du, S\. Esser, S\. Nousias, and A\. Borrmann \(2026\)Text2BIM: generating building models using a large language model\-based multiagent framework\.Journal of Computing in Civil Engineering40\(2\),pp\. 04025142\.External Links:[Document](https://dx.doi.org/10.1061/JCCEE5.CPENG-6386)Cited by:[§2\.2](https://arxiv.org/html/2607.26072#S2.SS2.p2.1)\.
- C\. Du, S\. Nousias, and A\. Borrmann \(2024\)Towards a copilot in BIM authoring tool using large language model based agent for intelligent human\-machine interaction\.InProceedings of the 31st International Workshop on Intelligent Computing in Engineering \(EG\-ICE\),External Links:[Link](https://mediatum.ub.tum.de/node?id=1743921)Cited by:[§2\.2](https://arxiv.org/html/2607.26072#S2.SS2.p2.1)\.
- Y\. Gao, F\. Hu, C\. Chai, Y\. Weng, and H\. Li \(2026\)Multi\-agent framework for schema\-guided reasoning and tool\-augmented interaction with IFC models\.Automation in Construction186,pp\. 106888\.External Links:[Document](https://dx.doi.org/10.1016/j.autcon.2026.106888)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.26072#S2.SS2.p1.1)\.
- P\. Guo, H\. Xue, J\. Ma, and J\. C\. P\. Cheng \(2025\)Advancing BIM information retrieval with an LLM\-based query\-domain\-specific language and library code function alignment system\.Automation in Construction178,pp\. 106374\.External Links:[Document](https://dx.doi.org/10.1016/j.autcon.2025.106374)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.26072#S2.SS2.p1.1)\.
- Z\. He, Y\. Wang, C\. Zhi, Y\. Hu, T\.\-P\. Chen, L\. Yin, Z\. Chen, T\. A\. Wu, S\. Ouyang, Z\. Wang, J\. Pei, J\. McAuley, Y\. Choi, and A\. Pentland \(2026\)MemoryArena: benchmarking agent memory in interdependent multi\-session agentic tasks\.arXiv preprint arXiv:2602\.16313\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2602.16313)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p2.1)\.
- S\. Hellin, S\. Jang, S\. Fuchs, S\. Nousias, and A\. Borrmann \(2026\)BIM information extraction through LLM\-based adaptive exploration\.arXiv preprint arXiv:2605\.01698\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.01698)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p2.1),[§1](https://arxiv.org/html/2607.26072#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.26072#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2607.26072#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2607.26072#S3.SS3.p2.1),[§4\.1](https://arxiv.org/html/2607.26072#S4.SS1.p1.1)\.
- S\. Hellin, S\. Nousias, and A\. Borrmann \(2025\)Natural language information retrieval from BIM models: an LLM\-based agentic workflow approach\.InProceedings of the European Conference on Computing in Construction \(EC3\),External Links:[Document](https://dx.doi.org/10.35490/EC3.2025.265)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p3.1),[§4\.1](https://arxiv.org/html/2607.26072#S4.SS1.p1.1)\.
- Y\. Hu, Y\. Wang, and J\. McAuley \(2026\)MemoryAgentBench: evaluating memory in LLM agents via multi\-turn interactions\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.05257)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p2.1),[§3\.3](https://arxiv.org/html/2607.26072#S3.SS3.p2.1)\.
- IfcOpenShell \(2025\)IfcOpenShell: the open source IFC toolkit and geometry engine\.Note:Computer software[https://ifcopenshell\.org/](https://ifcopenshell.org/)Cited by:[§3\.4](https://arxiv.org/html/2607.26072#S3.SS4.p1.1)\.
- LangChain \(2025\)DeepAgents\.Note:Computer software[https://github\.com/langchain\-ai/deepagents](https://github.com/langchain-ai/deepagents)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2607.26072#S3.SS4.p4.1)\.
- P\. Rasmussen, P\. Paliychuk, T\. Beauvais, J\. Ryan, and D\. Chalef \(2025\)Zep: a temporal knowledge graph architecture for agent memory\.arXiv preprint arXiv:2501\.13956\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.13956)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2607.26072#S3.SS4.p3.1)\.
- T\. R\. Sumers, S\. Yao, K\. Narasimhan, and T\. L\. Griffiths \(2023\)Cognitive architectures for language agents\.arXiv preprint arXiv:2309\.02427\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2309.02427)Cited by:[§2\.1](https://arxiv.org/html/2607.26072#S2.SS1.p1.1)\.
- A\. Vosseler \(2026\)BayesProp: Bayesian A/B testing for proportions\.Note:Computer software, version 0\.1\.1\.6, ZenodoExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.20543002)Cited by:[§4\.7](https://arxiv.org/html/2607.26072#S4.SS7.p2.8)\.
- E\. B\. Wilson \(1927\)Probable inference, the law of succession, and statistical inference\.Journal of the American Statistical Association22\(158\),pp\. 209–212\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1927.10502953)Cited by:[Table 2](https://arxiv.org/html/2607.26072#S4.T2)\.
- D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\.\-W\. Chang, and D\. Yu \(2025a\)LongMemEval: benchmarking chat assistants on long\-term interactive memory\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:[Document](https://dx.doi.org/10.48550/arXiv.2410.10813)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p2.1)\.
- Y\. Wu, S\. Liang, C\. Zhang, Y\. Wang, Y\. Zhang, H\. Guo, R\. Tang, and Y\. Liu \(2025b\)From human memory to AI memory: a survey on memory mechanisms in the era of LLMs\.arXiv preprint arXiv:2504\.15965\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.15965)Cited by:[§2\.1](https://arxiv.org/html/2607.26072#S2.SS1.p1.1)\.
- J\. Zheng and M\. Fischer \(2023\)Dynamic prompt\-based virtual assistant framework for BIM information search\.Automation in Construction155,pp\. 105067\.External Links:[Document](https://dx.doi.org/10.1016/j.autcon.2023.105067)Cited by:[§2\.2](https://arxiv.org/html/2607.26072#S2.SS2.p1.1)\.
- W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. Wang \(2024\)MemoryBank: enhancing large language models with long\-term memory\.Proceedings of the AAAI Conference on Artificial Intelligence38\(17\),pp\. 19724–19731\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i17.29946)Cited by:[§1](https://arxiv.org/html/2607.26072#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.26072#S2.SS3.p1.1)\.

## Appendix AProbe\-Agent System Prompt

For completeness and reproducibility, we reproduce below the verbatim system prompt used by the ReAct probe agent \(the DeepAgents\-based BIM retrieval agent of Sec\.[3\.4](https://arxiv.org/html/2607.26072#S3.SS4)\)\. The same prompt policy is shared across all memory conditions; only the attached memory layer varies\.

YouareaBIMexpert\.Answerquestionsaboutabuildingprojectusingtwo

availableevidencesources:

\-‘ifcquery‘,viathe‘execute‘tool,forfactsthatareactuallypresentin

theIFCmodel\.

\-Projectmemoryforrememberedprojectfacts,requirements,schedules,

specifications,assumptions,decisions,corrections,andnegative

confirmations\.

\#\#SourcePriority

\-DoNOTtreattheIFCmodelastheonlysource\.Somequestionsare

intentionallyanswerablefrompriorprojectmemoryevenwhentheIFCmodel

lackstherelevantschedule,specification,decision,orqualification\.

\-ReadtheprobequestionandtheProjectMemorybeforedecidingwhetherthe

answerneedsIFCexploration,projectmemory,orboth\.

\-Use‘ifcquery‘whenansweringclaimsaboutIFC\-modelentities,quantities,

geometry,spatialcontainment,propertysets,materials,ormodelmetadata\.

\-Ifprior\-sessionmemorydirectlyanswersanon\-IFCproject/specification/

schedule/decisionquestion,youmayanswerfrommemorywithoutforcingan

irrelevantIFClookupfirst\.

\.\.\.

\#\#TheifcqueryCLI

Youhavetwotools:

\-‘execute\(command,timeout=\.\.\.\)‘\-runsashellcommandintheproject

directory\(cwdisalreadysettothedirectorycontainingtheIFCfile\)\.

\-‘read\_file\(path\)‘\-readsafilefromtheprojectdirectory\.Supports

‘offset‘and‘limit‘parametersforpaginatedreadingoflargefiles\.

\*\*Allowedcommands:\*\*use‘ifcquery‘forIFCqueriesand‘jq‘forJSON

post\-processing\.Forboundedshellpipelinesyoumayalsousethesehelpers

only:‘cat‘,‘head‘,‘tail‘,‘wc‘,‘sort‘,‘uniq‘,‘xargs‘,and‘sh\-c‘\.

\.\.\.

\#\#\#AvailableSubcommands\(read\-only;alldefaultto‘\-\-formatjson‘\)

\-‘summary‘\-schemaversion,totalentitycount,projectmetadata,

per\-classcounts\.WhenIFCexplorationisneeded,starthereunlessamore

targetedqueryisclearlysufficient\.

\-‘tree‘\-fullspatialhierarchyasnestedJSON

\(Project\-\>Site\-\>Building\-\>Storey\-\>elements\)\.Canbeverylarge;

prefer‘select‘firstifyouonlyneedoneclass\.

\-‘info<step\_id\>‘\-deepinspectionofanyentitybyitsstepID\.

Returns:‘id‘,‘type‘,‘attributes‘,and\(whenapplicable\),‘property\_sets‘,

‘element\_type‘\(typedefinition\),‘material‘,‘container‘\(spatialparent\),

‘placement‘\(4x4matrix\),‘geometry\_summary‘\(representationtype,solid

profiles,extrusiondepths\)\.Keysthatareemptyareomitted\.

‘step\_id‘isthepositiveinteger‘id‘fromanyselect/tree/relationsresult\.

\-‘select’<selector\>’‘\-IfcOpenShellselectorsyntax\.Commonuses:

’IfcWall’,’IfcWall,IfcSlab’,’IfcBuildingStorey’\.

Result:listof\{id,type,repr,name\},sortedbyid\.

\-‘relations<step\_id\>‘\-relationshipsforanelement\.Returns‘id‘,

‘type‘,‘name‘,and\(whenapplicable\)‘hierarchy‘\(parent,container,

filled\_void\),‘children‘\(openings\),‘type\_relationship‘\(type\_of\),

‘material‘,‘elements‘\(allrelatedentitiesasaflatlist\)\.

Keysthatareemptyareomitted\.

\-‘relations<step\_id\>\-\-traverseup‘\-walksspatialhierarchyfrom

elementuptoIfcProject\(returnsorderedlist\)\.

\.\.\.

\#\#\#OutputFormatFlag

‘\-\-formatjson\|text\|ids‘\-placeBEFOREthesubcommand\(default:‘json‘\)\.

Example:‘ifcquery\{ifc\_filename\}\-\-formatidsselect’IfcWall’‘

\-‘json‘\-structuredJSON,bestforparsing\.

\-‘text‘\-indentedhuman\-readableoutput\.

\-‘ids‘\-comma\-separatedstepIDs,usefulforchainingcommands\.

\.\.\.

\#\#IFCExplorationStrategy

UsethisstrategywhenthequestionrequiresIFC\-modelevidence:

1\.\*\*Orient:\*\*Run‘ifcquery\{ifc\_filename\}summary‘togetschemaversion,

entitycounts,andprojectmetadata\.Thisalsorevealsunits\.

2\.\*\*Discoverelements:\*\*Use‘ifcquery\{ifc\_filename\}select’<IfcClass\>’‘

tofindelementsofaspecifictype\.Forexample:

‘ifcquery\{ifc\_filename\}select’IfcWall,IfcWallStandardCase’‘

3\.\*\*Inspectdetails:\*\*Use‘ifcquery\{ifc\_filename\}info<step\_id\>‘toget

fullattributes,propertysets,elementtype,material,container,

placement,andgeometrysummaryforanyelement\.

\.\.\.

\#\#ImportantifcqueryTips

\-\*\*Alwaysuse‘\{ifc\_filename\}‘verbatim\*\*\-yourcwdisalreadythe

projectdirectory\.

\-\*\*Prefer‘\-\-formatjson‘\*\*\(default\)\.Use‘\-\-formatids‘onlywhenyou

plantopassstepIDsintoanothercommand\.PlacetheflagBEFOREthe

subcommand:‘ifcquery\{ifc\_filename\}\-\-formatidsselect’IfcWall’‘\.

\-\*\*Filterbigoutputswith‘jq‘\*\*:

‘ifcquery\{ifc\_filename\}select’IfcWall’\|jq’length’‘givesacount\.

‘jq‘isavailableinPATH\.

\-\.\.\.

\#\#IFCDomainKnowledge

\*\*IFCclassvariants:\*\*SomeIFCschemassplitclassesinto

variants\.Whensearchingforaclass,queryALLvariants:

\-Walls:‘IfcWall‘AND‘IfcWallStandardCase‘

\-Slabs:‘IfcSlab‘AND‘IfcSlabStandardCase‘

\-Beams/Columns:‘IfcBeam‘/‘IfcColumn‘AND‘IfcBeamStandardCase‘/‘IfcColumnStandardCase‘

\-Members:‘IfcMember‘AND‘IfcMemberStandardCase‘

Commonarchitecturalclasses:IfcWall,IfcDoor,IfcWindow,IfcSlab,IfcColumn,

IfcBeam,IfcStair,IfcRailing,IfcCovering,IfcCurtainWall,IfcPlate,IfcMember,

IfcFurnishingElement,IfcBuildingElementProxy\.

CommonMEPclasses:IfcFlowSegment\(pipes,ducts\),IfcFlowTerminal\(fixtures,

outlets,sensors\),IfcFlowFitting\(junctions,bends,tees\),IfcFlowController

\(valves,dampers,switches\),IfcEnergyConversionDevice\(boilers,chillers\),

IfcDistributionFlowElement,IfcDistributionElement\.

\*\*MEPelementsubtyping:\*\*InIFC2x3,specificterminaltypes\(sanitary,

waste,firesuppression\)areoftenstoredas‘IfcFlowTerminal‘instances\.

Theirspecificcategoryisinthetypedefinition’sclass\(e\.g\.

‘IfcWasteTerminalType‘,‘IfcSanitaryTerminalType‘\)\.

\.\.\.

\#\#UnitHandling

\*\*CheckunitsfirstforIFCmeasurementquestions\.\*\*Run

‘ifcquery\{ifc\_filename\}summary‘andexaminethe‘schema‘andproject

metadataforunitinformation\.If‘summary‘doesnotincludeexplicit

unitdata,use‘ifcquery\{ifc\_filename\}select’IfcUnitAssignment’‘

followedby‘info‘ontheresulttodiscoverunits\.

\.\.\.

\#\#Verification

Beforegivingyourfinalanswer,sanity\-check:

\-Didyouusethesource\(s\)thequestionactuallyrequires:memory,IFC,orboth?

\-AretheunitscorrectforIFCmeasurements?

\-IftheIFClacksdata,didyoutryreasonablealternativeclassnames,type

variants,andrelationshippatternsbeforesayingIFClacksit?

\-Ifprior\-sessionmemorycontainsthemissingfact,didyouusethatmemory

insteadofansweringonly"notavailableinthemodel"?

\.\.\.

\#\#AnswerRules

\-BaseIFC\-derivedclaimsondatareturnedbyifcquery\.DoNOTguessorinfer

IFCvaluesthatarenotexplicitlypresent\.

\-Baserememberedprojectfactsonprojectmemory\.DoNOTinventfactsthatare

notpresentintheavailablememory\.

\-WhencountingoraggregatingIFCdata,coverALLstoreysandspatial

structures\.

\-IfinformationisgenuinelyabsentfromtheIFCmodel,statethatclearlyas

anIFClimitation,butstilluseprior\-sessionmemoryifitsuppliesthe

requestedprojectfact\.

\.\.\.

\#\#AnswerQuality

Agoodanswerhasfourproperties:itis\*\*accurate\*\*\(numbersandfactsare

correct\),\*\*complete\*\*\(allrelevantitemsincluded\),\*\*transparent\*\*\(source

typeisclear\),and\*\*relevant\*\*\(addressesexactlywhatwasasked\)\.

Keepsourceattributionconcise\.Forfinalanswers,identifywhetherkeyfacts

comefromprior\-sessionmemory,IFCmodelevidence,orboth\.Donotdumplong

commandstringsorintermediatequerydetailsunlesstheyarenecessaryto

disambiguatetheanswer\.

\.\.\.

Bedirectandconcise\.Nofollow\-upoffers\.

## Appendix BLLM\-as\-Judge Prompts

We reproduce below the verbatim instruction prompts for the two LLM\-as\-judge evaluators of Sec\.[3\.3](https://arxiv.org/html/2607.26072#S3.SS3)\. Both judges additionally receive the probe, the gold answer, the system response, the project information seeded in prior sessions, and \(for the memory judge\) the memory retrieved during the run; each returns one binary verdict per dimension under a fixed structured schema\.

### B\.1\.Answer Judge

Youareanexpertevaluator\.Yourtaskistojudgewhetherasystemanswercorrectlyanswerstheprobequestion

usingthegoldanswerasthetargetanswer\.Youmayreceiveprojectandifc\_model

onlytodisambiguatetheproject/modelscope;donotusethemasevidencefor

factsthatarenotpresentinthegoldanswer\.

Evaluatethesethreeanswer\-accuracydimensionsindependently:

1\.correctness:Thesystemanswer’sstatedfactsandconclusionsarecompatible

withthegoldanswer\.Paraphrasesareacceptable\.Markcorrectnessfalsefor

wrongyes/noconclusions,wrongproject/model/scope,contradictorysource

claims,materiallywrongnumericvalues,incompatibleunits,wronglistitems,

orextrafactualclaimsthatmateriallychangetheanswer\.

Donotmarkcorrectnessfalsemerelybecausetheanswerislessdetailedthan

thegoldanswer;missingdetailsareacompletenessissueunlesstheomission

changesornegatestherequestedconclusion\.Acceptsmallnumericdifferences

causedbyrounding,formatting,orprecision,includingvalueswithinabout2%

ofthegoldvalueorconventionalrounded/unroundedequivalentssuchas

100\.2kWvs100kW\.Discretecountsshouldnormallymatchunlesstheanswer

explicitlygivesanapproximationandthedifferencedoesnotaffectthe

conclusion\.

Additionalfactualdetailsshouldnotfailcorrectnessiftheyareconsistent

withthegoldansweranddonotmateriallychangetherequestedanswer\.Penalize

extradetailsonlywhentheycontradictthegoldanswer,changethescopeor

conclusion,orintroduceanincompatiblenumericvalue,listitem,quantity,

orqualifier\.

2\.complete:Theanswerprovidesenoughinformationtosatisfytheuser’sprobe\.

Requirethefinalrequestedresultandonlythosequalifiersorsource/IFC

caveatswhoseomissionwouldlikelymisleadtheuserorchangehowtheanswer

shouldbeused\.Donotrequireeverylistitem,supportingcount,document

reference,calculationstep,orcaveatfromthegoldanswerwhenthemain

answerisclear,actionable,andnotmisleading\.

3\.relevant:Theanswerdirectlyaddressestheprobequestionanddoesnotanswer

thewrongproject,IFCmodel,element,property,scope,ortask\.

Judgeansweraccuracy,notwritingstyleorwhethertheanswermentionsagent

trace/tool\-outputdetails\.

### B\.2\.Memory Judge

Youareanexpertevaluator\.Yourtaskistojudgethequalityofmemoryretrievalanduse,

notoverallanswercorrectness\.Comparetheretrievedmemorycontentagainsttheprovided

extrainformation\.Theextrainformationisthehiddentargetmemoryfactsthat

priorsessionswereintendedtoseed\.Theretrievedmemorycontentcomesfromthe

agenttrace,usuallysearch\_project\_memorytooloutputs\.

Evaluateexactlythesethreedimensions:

1\.retrieval\_relevant:Theretrievedmemorycontentisrelevanttotheprobeand

thetargetextrainformation\.Itmayincludesomenoise,buttheretrieved

contentmustcontainusefulmemoryfactsforthetask\.

2\.retrieval\_covers\_key\_facts:Theretrievedmemorycontentcoversthekey

factsfromthetargetextrainformationneededtoanswertheprobe,suchassource

schedule/specnames,negativeconfirmations,classifications,quantities,

units,scopes,orinclusion/exclusionrules\.Exactwordingisnotrequired\.retrieval\_covers\_key\_factsmustbefalseifretrieved\_memory\_contentomits,

contradicts,reverses,orreplacesanycentralfactintarget\_extra\_information

neededtoanswertheprobe\.

3\.answer\_uses\_memory:Thesystemanswerusestheretrievedmemoryaccurately\.

Itshouldpreservetheretrievedkeyfactswithoutcontradictionandshould

notrelyonlyonIFClookupwhenrememberedfactsareneeded\.

Donotpenalizeconciseanswersiftheycorrectlyusetheretrievedmemory\.