From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data

# From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data
Source: [https://arxiv.org/html/2607.16845](https://arxiv.org/html/2607.16845)
###### Abstract

Scientists at European XFEL conduct experiments that generate very large and complex datasets\. The subsequent data analysis is challenging as scientists must combine their domain expertise with facility\- and software\-specific knowledge scattered across documentation, tools, and support channels\. To address this problem, we designed and evaluated an agenticartificial intelligence \(AI\)system tailored to the scientists’ needs and integrated with the high\-performance computing environment of European XFEL\. Using a design science research approach, we conducted a rapid literature review, a systematic evaluation of 16AItools, multiple interviews, a focus group, and a user study with experts at European XFEL to develop and evaluate two prototypes\. Our study identifies key knowledge challenges in scientific data analysis, derives requirements for anAIagent that supports knowledge retrieval and source code generation, and proposes design recommendations for a specialized system adaptable to the evolvingAItool landscape\. These findings provide guidance for developing maintainableAIsupport in highly specialized scientific environments\.

## IIntroduction

European XFEL is auser facilitywhere visiting scientists \(users\) exploit unique types of X\-ray pulses to study matter on multiple, complex instruments\[decking\_mhz\-repetition\-rate\_2020\]\. These users bring strong domain expertise, but they often need help navigating the facility\-specific analysis environment and translating experimental data into scientific findings\.

The experiments produce extremely large and complex datasets, some on the petabyte scale\[malka\_data\_2023,sobolev\_data\_2024\]\. To turn this data into scientific results, users need significant computing resources and ad\-hoc data analysis software that fit the experimental context\. Like similar facilities, European XFEL enables this process through ahigh\-performance computing \(HPC\)cluster, specialized software, and user support provided by staff with expertise in photon science, data science, and software engineering\.

The user support is distributed across extensive documentation and direct assistance, which makes it strong and comprehensive\. However, this support does not scale well, as the level of direct assistance is constrained by available human resources, while static documentation lacks the contextual guidance required for the broad range of concrete analysis tasks\. These limitations are further amplified by the diversity of users’ data analysis experience and needs, and even more so considering thatoffline data analysiscan span several months following data collection\[schmidt\_turning\_2024\]\.

To improve the current situation, users need a context\-sensitive, always available system that bridges the gap between general knowledge resources and their individual workflows\[maalej\_task\-first\_2009\]\.AI\-based support systems can narrow this divide by turning the existing bulk of documentation into contextualized assistance\.Retrieval\-augmented generation \(RAG\)is an especially relevant concept because it helps the assistant to ground responses in curated knowledge and reducehallucinations\[james\_retrieval\-augmented\_2025\]\.AIagents extend this support by not only answering questions but also planning and executing tasks, including code generation\[sapkota\_ai\_2026\]\. However, for scientific workflows, an agent is only useful if the system behavior remains traceable, controllable, and compatible with human oversight\[tang\_risks\_2025\]\.

In this work, we designed and evaluated an agenticAIsystem tailored to offline data analysis at European XFEL\. The system is intended to help users retrieve documentation and generate analysis code while being integrated with theHPCcluster\. Our aim was to provide a powerful assistant that is always available and context\-sensitive to augment the knowledge resources of the facility\[maalej\_task\-first\_2009,maalej\_lightweight\_2008\]\.

We followed adesign science research \(DSR\)approach with seven activities to answer the following research questions:

RQ1:What knowledge challenges do facility users and members encounter during offline data analysis?

RQ2:What requirements must an agenticAIsystem meet to address the challenges?

RQ3:What design recommendations for effective and maintainableAIsupport can be derived from deploying and evaluating the system in practice?

The remainder of the paper presents the methodology and findings, the discussion, the limitations, and related work\.

## IIDesign Science Research Approach

### II\-AOverview

TABLE I:Research activities aligned with the research questions and the design science research approach bypeffers\_design\_2007RQ1RQ2RQ3No\.ActivityPIOSDDDEEV1Interviews with 2 managers to discuss the study scope and initial tool concept\.XX2Focus group with 16 Data Analysis members to analyze workflow and challenges in offline data analysis\.X3Interviews with 3 facility users and 5 members to analyze offline data analysis challenges and requirements for anAIagent\. Debriefing with 13 Data Analysis members\.XX4Design/demonstration first prototype\. Evaluation with 12 Data Analysis members\. Requirements reassessment\.XXXX5Rapid multivocal review of 224 articles to integrate insights from related studies into the requirements\.X6Evaluation of solutions for the agentic AI system components\. Reassessment of requirements\.XX7Design/demonstration of second prototype\. Evaluation with 13 experts\. Iterative prototype improvement\.XXXX
PI = problem identification and motivation, OS = definition of the objectives for a solution, DD = design and development, DE = demonstration, EV = evaluation, Step 6 = communication \(not listed above; we communicate our findings in this paper\)

DSRis appropriate for this exploratory study because it focuses on designing and evaluating IT artifacts for organizational problems\[peffers\_design\_2007\]\. This study proceeds through iterative cycles of problem identification, requirements engineering, tool design, and evaluation\.DSRmakes the methodological logic of these activities explicit\[engstrom\_how\_2020\]\.

[Table I](https://arxiv.org/html/2607.16845#S2.T1)maps our research activities to the sixDSRsteps according topeffers\_design\_2007, as well as to our research questions\. We conducted seven activities, each aligned with at least oneDSRstep\. The process was iterative: the insights gained from the first prototype informed the requirements analysis, design, and evaluation of the second prototype \([Figure 2](https://arxiv.org/html/2607.16845#S2.F2)\)\. The following subsections describe the activities in chronological order\. Supporting material, such as interview minutes, detailed software evaluation artifacts, and the prototype setup, is available in thereplication package\[fuchs\_reppack\_2026\]\.

### II\-BInitial Interviews with Managers

The first author conducted five semi\-structured interviews with the two managers of the Data Analysis and Controls groups at European XFEL between June and December 2025\. The objectives were to iteratively scope the challenges during offline data analysis and to identify initial design objectives for a solution\. We selected both managers as participants of this exploratory first activity because their roles place them close to the software, infrastructure, and user support for data analysis, as well as toAIinitiatives at the facility\.

The interviews showed that support requests of users concentrate on three phases: the early phase of experiment preparation, the experiment execution phase, including online \(real\-time\) data analysis, and the late phase of offline data analysis\. In all phases, users must combine their scientific domain expertise with facility\-specific knowledge about data formats, instrument settings, and theHPCenvironment\. The managers also emphasized that the existing software manuals of the facility are comprehensive but may be too broad to guide concrete analysis tasks effectively\. This is why the Data Analysis group started to move toward more experiment\- and technique\-oriented documentation recently\[euxfel\_fxe\_2026\]\.

When the discussion turned to solution objectives, the managers converged on the need for first\-level support during offline data analysis\. They saw value in a system that can retrieve relevant documentation, provide tailored Jupyter notebook templates, help users navigate complex datasets, optimize analysis code for theHPCenvironment, and report agent conversations to staff to improve traceability\. These capabilities were seen as especially relevant outside normal support hours, when users still need guidance but staff availability is limited\.

The interviews also left one important design question unresolved: whether the system should primarily support beginners, experienced users, or both\. We therefore treated these initial interviews as scoping activities and used the results to inform the subsequent focus group and interviews with facility users and members, where we refined the requirements and narrowed the prototype design\.

### II\-CFocus Group with Data Analysis Group

Our objectives for the focus group were to characterize the current offline data analysis workflow at European XFEL and to identify the knowledge\-related challenges within this process\. We chose a focus group because the participants shared the same domain and could build upon one another’s experience\[kontio\_focus\_2008\]\. The 60\-minute session brought together 16 members of the Data Analysis group in January 2026 and was moderated by the first author\. We used a digital mind map to take notes transparently\[luke\_improving\_2014\]and shared the mind map with participants after the session to support member checking\[burgessallen\_using\_2010\]\.

The discussion showed that offline data analysis is highly variable across scientific instruments and experiments, but it still follows a recurring pattern\. In broad terms, the work starts with data acquisition and data transfer to the offline storage of theHPCenvironment, continues with the actual data analysis, and ends with scientific findings\. During these steps, users inspect experiment logs\[maia\_mylog\_2025,maia\_mymdc\_2025\], exchange with staff, set up an analysis environment, and develop analysis scripts based on templates, libraries, and other software relevant to the experiment\. A typical setup consists of a JupyterLab server on theHPCcluster\[reppin\_interactive\_2021\], a software module with multiple libraries prepared by the Data Analysis group, and further user\-provided software\.

A second finding was that relevant knowledge is fragmented across many sources\. Participants pointed to general user documentation of the Data Analysis group, library documentation, instrument\-specific resources, technical reports, publications \(e\.g\., byturkot\_extra\-xwiz\_2023\), experiment proposals, and documentation of theHPCcluster\. They emphasized that the main challenge is not the existence of these different resources, but the need to combine them into a workflow for a concrete analysis objective\. This is the point at which anAIsystem could add significant value by retrieving and synthesizing relevant documentation rather than merely listing it\[james\_retrieval\-augmented\_2025\]\.

TABLE II:\(RQ1\) challenges in offline data analysis at European XFELNo\.ChallengesUsers:1Clarity of data analysis objectives2Mismatch between expected and actual data analysis complexity3Limited facility support during offline data analysis phase4Unfamiliarity with infrastructure complexity5Unfamiliarity with high\-performance computing6Unfamiliarity with data analysis procedures of instruments7Retrieval and synthesis of scattered documentationMembers of Data Analysis group:8Unfamiliarity with data analysis procedures of instruments9Retrieval and synthesis of scattered documentation10Understanding of analysis objectives of users11Time effort for user supportStaff scientists:12Understanding of analysis objectives of users13Time effort for user supportOverall, the focus group revealed three overlapping sets of 13 challenges, which are summarized in[Table II](https://arxiv.org/html/2607.16845#S2.T2)\. Users may struggle with concretizing their analysis objectives and steps, understanding the infrastructure,HPCenvironment, and analysis procedures, and synthesizing the scattered documentation\. Members of the Data Analysis group face the same documentation challenge, but they also need to infer users’ analysis objectives and spend time supporting them\. Staff scientists experience a similar support burden, especially when objectives are unclear\.

The findings established the problem space related to the first research question and informed the subsequent interviews with facility users and members, where we refined the challenge set and translated it into requirements\.

### II\-DInterviews with Facility Users and Members

We conducted informal interviews during the poster session of the annual European XFEL and DESY Users’ Meeting in January 2026\. Our objective was to capture firsthand experiences with knowledge challenges in offline data analysis and to explore expectations for anAIassistant\. The first author spoke with three facility users and five members, including four staff scientists and one manager\. All participants had experience with offline data analysis and the available user support\. Despite the limited participation, the interview notes and the subsequent debriefing with 13 members of the Data Analysis group helped to refine the insights regarding problem identification and solution objectives\.

Two members emphasized that staff scientists and users highly value all support from the Data Analysis group\. However, one user noted that reading documentation is tedious\. This comment reinforces the challenge of retrieving and synthesizing scattered documentation into a usable workflow, which the earlier focus group had already identified\.

The interviewees were most engaged when discussing possible features of anAIassistant for data analysis\. The features they considered most valuable were support for technical questions about facility software and infrastructure, plotting complex data, and optimizing code forHPCexecution\. They also emphasized that the system must not send data to third parties\. More optional feature suggestions included adapting the style of generated code to user preferences and providing daily summaries of experiment logs\. Finally, during the debriefing, the Data Analysis group identified a new risk due toAIassistance: an agentic system could increase the demand for human support if users begin reporting issues withAI\-generated code\.

These interviews broadened our analysis by adding views from users and staff scientists\. They confirmed the challenges mentioned by the Data Analysis group and provided design objectives for the first prototype, which we designed next\.

### II\-EDesign, Demonstration, and Evaluation of First Prototype

We designed and evaluated the first prototype in early March 2026 as a proof of concept for first\-level user support during offline data analysis\. To align the prototype with the typical analysis setup, we selected Jupyter AI as the user interface, which integrates directly into Jupyter Lab\. The prototype used the built\-in chat persona of Jupyter AI and was connected to aRAGbackend, which was internally developed at European XFEL\. This architecture enabled responses grounded exclusively in curated documentation\.

We deployed this system on a local computer and connected Jupyter Lab to a remote Jupyter kernel exposed by theHPCcluster\. This configuration gave theAIagent access to the data, software, and computational resources needed for offline analysis\. We used GPT\-5\.4\-mini as the chat completion model and text\-embedding\-3\-large for document embedding and knowledge retrieval\. At the time, both were current OpenAI models\. We limited the knowledge base to the documentation repository of EXtra\-data, the European XFEL library for reading datasets stored on theHPCcluster\. This setup was sufficient for a first demonstration of the solution concept\.

We evaluated the prototype in a dedicated meeting with 12 members of the Data Analysis group\. The first author instructed the system to generate a Jupyter notebook for offline data analysis\. The resulting notebook contained code to import EXtra\-data, load a dataset, and produce a data visualization\. Participants perceived Jupyter AI as a straightforward user interface, indicating that the familiar notebook environment can lower the barrier toAIadoption\[mcnutt\_design\_2023\]\.

The evaluation also exposed limitations of the prototype\. The system lacked a sophisticated agent loop to test and repair code autonomously\[sapkota\_ai\_2026\]\. It generated code that we could not fully execute because it contained incorrect library calls\. The participants hypothesized that thesehallucinationsstemmed from a discrepancy between our instructions and the information available in the knowledge base\. Had we granted the system access to the EXtra\-data GitHub repository, it might have retrieved additional documentation relevant to our request\. These observations led us to conclude that the next prototype iteration needed better context selection and automatic code verification\. Furthermore, it should support explicit human approval for code execution on theHPCcluster to prevent risky autonomous behavior by the agent\.

Overall, the first prototype showed that a familiar notebook\-based interface is a plausible entry point forAIsupport during offline data analysis\. However, knowledge retrieval alone is not sufficient for reliable code generation\. This feedback motivated a more comprehensive requirements analysis, which we began with a literature review\.

### II\-FRapid, Multivocal Literature Review

We conducted a rapid, multivocal literature review to extend our empirically derived requirements with further evidence from scientific and practitioner sources\. Because the study aims to inform a practical prototype rather than to produce an exhaustive literature review, we included second\- and third\-tier grey literature in addition to peer\-reviewed publications and followed established guidelines for systematic, rapid, and grey literature reviews\[kitchenham\_guidelines\_2007,ganann\_expediting\_2010,garousi\_guidelines\_2019,tugwell\_prisma\_2021\]\. Our review question focused on collecting requirements for agenticAIsystems that support scientists in creating source code for data analysis\.

We searched Google, Google Scholar, and Consensus AI, and supplemented the search with publications already known to us\. The search terms targeted human\-AIcollaboration, developer experience, andAIfor science\. We limited the publication window to January 2023 through March 2026 to focus onlarge language model \(LLM\)\-based agents\. The process yielded 224 relevant documents, including 141 peer\-reviewed articles, 46 preprints, 36 blog posts, and 1 report\.

To accelerate the thematic analysis, we used theLLMsGPT\-5\.4\-mini and MiniMax\-M2\.5 as judges and annotators\[wagner\_towards\_2025\]\. The two models annotated 3062 and 2303 candidate requirements, respectively\. Subsequently, GPT\-5\.4\-mini synthesized these candidates into 26 themes\. We manually validated and refined these results into 17 themes, clustered into requirements for system architecture, general agent behavior, knowledge retrieval, code generation, and optional features\.

The main trade\-off of a rapid review is that it favors speed over exhaustive coverage\[ganann\_expediting\_2010\]\. We did not perform snowballing, and the search strategy was intentionally narrow to align the review with the schedule for the second prototype\. In addition,LLMscan lead to variability in the results\[ernst\_genai\_2026\]\. We reduced this risk by using two models of different providers, requiring each extracted requirement to be paired with a text quote, and manually validating and reworking the results\.

TABLE III:\(RQ2\) requirements for the agentic AI systemNo\.The agentic AI system shall …System architecture:1Access the high\-performance computing cluster2Optimize latency during request processing\[yamada\_ai\_2025,schmidgall\_agentrxiv\_2025\]3Not share user data with third parties\[bano\_qualitative\_2025\]4Require minimal implementation effort5Require minimal effort to replace componentsGeneral agent behavior:6Interact with content of computational notebook\[ramasamy\_ai\_2025\]7Decompose a request into verifiable goals\[gu\_how\_2024,tufano\_autodev\_2024\]8Request human approval for sensitive actions\[tang\_risks\_2025,tufano\_autodev\_2024\]9Close task with narrative summary and recommended steps\[fluker\_preparing\_2025\]10Use concise languageKnowledge retrieval:11Retrieve up\-to\-date information\[wang\_code\_2025,wehr\_virtuous\_2025\]12Access relevant context information within a project\[gao\_empowering\_2024,wang\_code\_2025\]13Provide and verify citations\[olivares\_large\_2025,wang\_scientific\_2023\]Code generation:14Generate code optimized for high\-performance computing\[dong\_survey\_2025,rasheed\_ai\-powered\_2024\]15Test code for correctness and safety risks\[ren\_towards\_2025,fakhoury\_exploring\_2024\]16Request user feedback to guide and improve solutions\[tufano\_autodev\_2024,herrera\_co\-explainers\_2026\]17Adapt code style and explanations to the user’s expectations\[pudari\_copilot\_2023,corso\_generating\_2024\]18Generate code documentation on project\- and code\-level\[nejjar\_llms\_2025,kruse\_can\_2024\]19Recommend using a version control system\[kazemitabaar\_improving\_2024\]Optional requirements:20Report conversations between user and agent to staff\[tufano\_autodev\_2024,samadi\_ai\_2024\]21Assist in drafting scientific manuscripts\[yamada\_ai\_2025,fuente\-ballesteros\_artificial\_2025\][Table III](https://arxiv.org/html/2607.16845#S2.T3)lists all requirements that the proposed agenticAIsystem for offline data analysis must fulfill\. Requirements \[2\-3\], \[6\-9\], and \[11\-21\] are \(refined\) results of the literature review\. The results were sufficient to guide the subsequent evaluation of tool solutions for the second prototype\.

### II\-GEvaluation of Solutions for Agentic AI System Components

![Refer to caption](https://arxiv.org/html/2607.16845v1/x1.png)Figure 1:Agentic AI system components and recommended solutions per componentThe rapidly evolvingAIlandscape offers many highly popular agenticAItools and extensions\. Based on the learnings from the evaluation of our first, internally developed prototype, we wanted to create a second, state\-of\-the\-art prototype that could be adapted and maintained in the long term by members and users of European XFEL\. We therefore shifted our focus to the evaluation of off\-the\-shelf components and added minimal effort for implementation and replacement to the previously analyzed requirements \(requirements 4 and 5 in[Table III](https://arxiv.org/html/2607.16845#S2.T3)\)\.

We decomposed the system into four components: an agentic tool, connectivity to theHPCcluster, connectivity to multiple knowledge sources, and reporting\. In March 2026, we evaluated a set of free, open\-source tools for each component\. These tools can be executed without external telemetry to meet the privacy\-oriented requirement\.[Figure 1](https://arxiv.org/html/2607.16845#S2.F1)provides an overview of the components and recommended solutions\.

The most critical decision concerned the connection to theHPCcluster\. We selected and evaluated theapplication programming interface \(API\)of Jupyter Lab, the Jupyter MCP Server, and Visual Studio Code Remote Development as possible solutions\. The Jupyter LabAPIwas the most stable option for our environment because it connected a local notebook file to a remote Jupyter kernel of the cluster while giving an agentic tool read, write, and execution access to the file\. The Jupyter MCP Server was a good agentic solution in principle, but we could not use it in combination with theHPCcluster because its dependencies did not meet the configuration of the cluster\. Visual Studio Code Remote Development also proved unreliable for our use case because of interrupted connections and long latencies\. Therefore, we favored the Jupyter LabAPIin combination with Visual Studio Code to connect to the cluster\.

For the agentic tool, Kilo Code, OpenCode, and goose were the most promising options to us because they supported localLLMs, theModel Context Protocol \(MCP\)for connectivity with other systems, parallel\-processed sub\-agent workflows, and natural\-language customization through mechanisms such asAGENTS\.mdand skill files\. Jupyter AI and the DeepAgents framework were less suitable\. Jupyter AI offered fewer flexible customization options and required installation on the Jupyter Lab server, which was not reliably possible in theHPCenvironment\. DeepAgents would have required building a custom agentic system from scratch\. We selected the Visual Studio Code extension Kilo Code for our second prototype because it aligned best with the Jupyter LabAPIsolution\.

For knowledge access, we connected the agentic tool to GitHub, the European XFEL GitLab server, and theRAGsystem Grounded Docs\. The agent accessed these sources throughMCPservers\. GitHub was the simplest case because the officialMCPserver provided a stable interface to public and private repositories\. GitLab required a third\-party solution, since the official version depended on configuration changes that regular users could not make on the European XFEL server\. For knowledge retrieval, Grounded Docs was preferable to OpenRAG because it remained lightweight and quickly returned relevant documentation chunks that the agent could synthesize into its own response\. In comparison, OpenRAG is a more heavyweight, standalone system that returns an already synthesized natural\-language response via itsMCPextension\. Additionally to theMCPservers, we allowed the agent to use its integrated feature for retrieving internet search results\.

Reporting agent conversations to staff members is a feature that the managers originally identified\. The agent should send reports to the Zulip messenger, which is a popular tool at European XFEL\. We selected a third\-partyMCPserver that enabled sending messages via a chatbot persona\.

This tool evaluation yielded a second prototype built around Kilo Code, the Jupyter LabAPI, andMCPconnections to GitHub, GitLab, Grounded Docs, and Zulip\. Although we did not design the evaluation process as a quantitative benchmark study, it was systematic and tailored to our iterativeDSRapproach\. The combination of the selected off\-the\-shelf solutions met all previously analyzed system requirements, which we demonstrate next\.

### II\-HDesign, Demonstration, Evaluation of Second Prototype

![Refer to caption](https://arxiv.org/html/2607.16845v1/assets/figures/prototype.png)Figure 2:Data analysis in Visual Studio Code with Kilo Code agent, agent customization files, and data and Jupyter kernel from HPC clusterTABLE IV:\(RQ3\) design recommendations for the agentic AI systemNo\.Design recommendationSystem architecture:1Decide on system architecture based on variance of user workflows\.2Check user’s tool expectations before recommending agentic tool\.General agent behavior:3Use programmatic hooks to force the agent to adhere to behavior\.4Configure required types of human approvals in advance\.Knowledge retrieval:5Make available documentation agent\-ready\.6Steer agent toward relevant knowledge sources\.7Force order of knowledge sources during knowledge retrieval\.8Enable user to customize knowledge retrieval workflow\.9Use small LLM with medium/high reasoning for knowledge retrieval\.Code generation:10Force spec\-driven development\.11Force iterative prototyping of minimal solutions\.12Forbid hard\-coding default values for mandatory user input\.13Force request to clarify the execution environment\.14Use large LLM with low/medium reasoning for code generation\.Based on the requirements analysis and components evaluation, we designed a second prototype based on the selected off\-the\-shelf tools \([Figure 2](https://arxiv.org/html/2607.16845#S2.F2)\), demonstrated its applicability for offline data analysis, and evaluated and iteratively improved the prototype during a user study in April 2026\. Afterward, we derived 14 design recommendations for maintainableAIsupport \([Table IV](https://arxiv.org/html/2607.16845#S2.T4)\)\.

A major part of the design process was the adjustment of agent behavior\. We used an AGENTS\.md file to explain the prepared code project, the expected data analysis workflow, and the behavioral rules to the agent\. The rules corresponded to the system requirements 6 to 19 \([Table III](https://arxiv.org/html/2607.16845#S2.T3)\)\. We added requirement 10 \(use concise language\) after encountering verbose responses from the agent during initial experiments\. To transform these responses into a concise, action\-oriented list of bullet points, we instructed the agent to use thecavemanskill \(in lite mode\), which was a popular skill at that time\[brussee\_caveman\_2026\]\.

We evaluated the prototype in 13 one\-hour sessions between the first author and individual European XFEL members\. These included eight members with an occupation in data science or computer science, as well as five with an occupation in photon science\. Participants brought their own unique use cases, ranging from visualizing data and analyzing specific data aspects to preparing batch jobs for theHPCcluster\.

Participants appreciated the agent behavior and responsiveness, the quality of knowledge retrieval results and generated source code, and the ability to execute Jupyter notebooks on theHPCcluster\. Furthermore, they considered the reporting feature useful for accelerating support processes\.

\#\#ExecuteandTestCodeonHPCCluster

Alwaysfollowthesestepsafteryougeneratedcode:

1\.DONOTtestthecodebyexecutingitinyour

internalsandbox\.Instead,createanewcode

fileoreditanexistingone\(e\.g\.,aJupyter

notebook\)andexecutethefilewithanavailable

JupyterkernelorPythonenvironment\.

\-ALWAYSasktheuserfirst,whichJupyterkernel

orPythonenvironmentyoushouldusetoexecute

thecode\.

\-YouMUSTNOTexecutethecodeinyourinternal

sandboxasthesandboxdoesnothaveaccessto

thedataandcomputationalenvironmentofthe

HPCcluster\.

2\.Reviewifanyerrorsappeared\(e\.g\.,error

messagesinnotebookcelloutput\)\.

3\.Iferrorsappeared,reworkyourcodeandstart

againwithstep1\.

Listing 1:AGENTS\.md instructions for code execution on HPC clusterThe evaluation also exposed recurring friction points\. We had to force the agent to execute, test, and refine code directly within the notebook rather than in its internal sandbox, so that it could use the data, libraries, and computational resources of the cluster \([1](https://arxiv.org/html/2607.16845#LST1)\)\. When we used smallLLMsfor chat completions, the agent sometimes failed to reuse the code of European XFEL libraries\. Additionally, the system also produced many explicit approval requests during the first sessions, including for uncritical read actions\. These observations led us to refine the prototype iteratively\.

We updated the prototype during and after each session\. Typical updates during a session involved adding documentation to Grounded Docs to accommodate the specific use cases of participants\. After the sessions, we updated the instructions in the AGENTS\.md file if the agent did not behave as the participant or we had expected\. Overall, we found natural\-language instructions not sufficient to ensure stable behavior\. The agent reacted differently depending on the selectedLLM, reasoning level, and wording\. Programmablehookswould constitute more reliable options for instructions\. At that time, however, Kilo Code did not yet support these hooks\. Instead, it supported explicit rule files, which we tested during one session\. However, they were harder to maintain and less interoperable than an AGENTS\.md file, and did not lead to improved agent behavior\.

The 14 design recommendations emphasize, for example, the need for configurable system components tailored to user needs, explicit human approval for \(exclusively\) sensitive actions, programmable hooks to complement natural\-language instructions, and agent\-compatible documentation\. Overall, the second prototype demonstrated that a selection of off\-the\-shelf tools can be adapted to the offline data analysis workflows at European XFEL\. However, this was possible only through continuous design iterations\.

### II\-IAnswers to the Research Questions

Our answers to the research questions of this study characterize the challenges of offline data analysis at European XFEL, as well as the requirements and design recommendations for an agenticAIsystem that addresses the challenges\.

Answering RQ1,we found 13 partially overlapping knowledge challenges of facility users, Data Analysis group members, and staff scientists during offline data analysis at European XFEL \([Table II](https://arxiv.org/html/2607.16845#S2.T2)\)\. The key challenge is that users must turn scattered and partially tacit knowledge into a project\-specific data analysis workflow\. The difficulty is not only clarifying analysis objectives and understanding the complex infrastructure and software, but also synthesizing various documentation resources and informal support\. Therefore, the fundamental problem in practice is the integration and contextualization of knowledge\[robillard\_\-demand\_2017\], and not necessarily the production of documentation\.

Answering RQ2,we analyzed 21 requirements that translate the challenges into concrete design objectives for an agenticAIsystem \([Table III](https://arxiv.org/html/2607.16845#S2.T3)\)\. A useful system must be integrated with theHPCcluster, support both knowledge retrieval and code generation, provide verified citations and code testing, and enforce human oversight for sensitive actions\. The additional requirements for minimal implementation and replacement effort are especially important for this study because they capture the need for a maintainable system that can adapt to an evolving tool landscape\.

Answering RQ3,we derived 14 design recommendations that show how the requirements can be implemented in practice \([Table IV](https://arxiv.org/html/2607.16845#S2.T4)\)\. They suggest that maintainableAIsupport depends on the right balance between flexibility and control: off\-the\-shelf tools are preferable to custom\-built systems, but only when they can be configured to match required user workflows and safety constraints\. The recommendations also reinforce two central lessons of the study: documentation must be agent\-ready, and behavioral control should rely on programmable hooks rather than on natural\-language instructions alone\.

Overall, the answers to the research questions support the main objective of the study: to move scientists from overload to insights by designing an agenticAIsystem that can integrate knowledge retrieval, code generation, and theHPCenvironment in a maintainable way\. The findings show that this is possible, but only if the system is grounded in the user workflow, controlled carefully, and built from components that can evolve with the surrounding tool ecosystem\.

## IIIDiscussion

### III\-AAI Agents Become Standard Workflow Components

Our findings suggest thatAIsupport is becoming a normal part of the workflows at European XFEL\. This trend becomes particularly evident when we compare the currentAIusage of the user study participants with an earlier study conducted at the facility\[kruse\_can\_2024\]\. What started as occasional assistance for knowledge retrieval is increasingly moving toward agents that are embedded in everyday coding and analysis work\.

Available tools are already sufficient to support different user needs and preferences\. Our tool analysis showed that the combination of a local agentic tool andMCPservers, including a lightweightRAGservice, can cover the requirements of offline data analysis\. A working system can be configured by a facility user without changes to the existing infrastructure and software, which makes practical adoption feasible\.

At the same time, broader adoption of agentic tools at European XFEL still depends on careful testing, clear guidance, and management support\. Beyond technical feasibility, the remaining challenges are safe operation, robustness, and helping users configure the tools in ways that fit their workflows\. In that sense,AIagents should be treated less as add\-ons and more as emerging, soon indispensable workflow components that need deliberate integration into the facility environment\.

### III\-BMaintainable AI Systems Require Flexible Components

Our analysis suggests that maintainability depends less on reducing the number of system parts than on keeping those parts modular and replaceable\. The second prototype was more complex than the first one, but it was also easier to maintain because the agentic core, the knowledge source connectors, and the reporting mechanism could be used out of the box and configured independently\. In the European XFEL setting, a more elaborate configuration can be more sustainable than a self\-developed solution if each component can be swapped or adjusted without rebuilding the whole system\.

The tool evaluation showed that several technically feasible combinations of components are available, but only a few are likely to fit everyday work at European XFEL\. Users could connect to theHPCcluster through local or remote Visual Studio Code sessions, the browser\-based Jupyter Lab, or programmaticMCPinterfaces\. Furthermore, some agentic tools expose multiple frontends, including command\-line interfaces, standalone desktop apps, and web apps\. Theoretically, users could use the OpenCode web interface to orchestrate the data analysis process from their smartphones\. In practice, however, the most relevant options are a local Visual Studio Code window, connected to the Jupyter LabAPIand Kilo Code, or a similar setup based on a remote SSH connection\.

Another maintainability aspect concerns behavioral control\. Our user study showed that AGENTS\.md and skill files are useful for describing workflows, but they do not produce stable behavior across different models, reasoning levels, and instruction formulations\. Reliable workflows, therefore, need programmable hooks rather than only natural\-language instructions, especially when tasks such as code commits or other traceable actions should happen reliably\. As tools such as OpenCode and Claude Code already support hooks, we expect that this will become a standard feature soon\. At the same time, natural\-language configuration options remain valuable for rules that are easier to express and maintain in prose\. Because the landscape ofAItools continues to evolve rapidly, a maintainable agenticAIsystem should apply interoperable configuration options that enable a low\-effort transition to new system components and protocols\.

### III\-CAgents Require AI\-First Documentation

If European XFEL users and members increasingly rely on agents to retrieve knowledge, future\-proof technical documentation must be designed forAIconsumption\. That means the documentation must be structured, complete, and available in lightweight formats that agents can process reliably\.

Our user study suggests thatAPIreference documentation alone is not sufficient\. Participants often valued the generated code, but they also observed that the agent missed library\-specific details when the available documentation was incomplete or too superficial\. This matches the move of the Data Analysis group toward cookbook\-style guidelines for selected scientific instruments and experiment types, because workflow\-focused examples give humans andAIagents the context they need to produce correct data analysis code\.

The paradigm shift towardAI\-first documentation resembles mobile\-first web design, in which smartphone screens determine the structure of modern websites\. For documentation, that implies maintaining an agent\-consumable format alongside or instead of the existing resources\. One practical option for websites is adding a Markdown\-basedllms\.txtfile\[howard\_llmstxt\_2024\]\. Such a layer can reduce parsing problems and lower the retrieval overhead of web scrapers\. Its value, however, depends on consistent maintenance by documentation owners and on scrapers that actually focus on the simplified format\[mueller\_crawlers\_2026\]\.

### III\-DEthical Aspects Remain Essential for AI Acceptance

Multiple research activities of this study showed that ethical aspects are essential for the acceptance ofAIsupport in complex scientific environments\. In our setting, the primary concerns centered on traceability of retrieved information, transparency of internal agent workflows, and human oversight regarding sensitive actions\. Study participants were willing to rely on the agent when it cited its sources, explained what it was doing, and left room for user intervention\.

Citations played an important role in building trust during knowledge retrieval\. Participants could trace the sources of the response through the linked references in the agent output and check whether the cited material supported the recommendation\. This made the retrieval process more inspectable and helped distinguish grounded answers from unsupported ones\.

Implementation plans were equally important because they made the agentic reasoning process easier to review\. By showing a checklist of steps before execution, the system allowed users to inspect the planned behavior and adjust it if needed\. This was especially relevant in our scientific setting, where the generated code directly affects the results of scientific analyses\.

Human approval remained necessary for actions that interact with the realHPCenvironment\. The prototype had to execute and test code against actual data, libraries, and compute resources\. This creates risks if the agent would act without oversight\. The explicit approval requests had to focus on code edits and execution rather than on read operations to ensure that human attention was directed toward critical decisions\.

These observations suggest that ethical guidance for agentic systems should not remain in theoretical guidelines\. A promising research approach involves adapting available taxonomies of ethicalAIrequirements\[puhlfurs\_model\_2025\]to agentic systems and executing recurring ethical reviews using an agentic skill file\. Such a skill would bring ethical requirements closer to modernAIworkflows and facilitate their implementation\.

## IVLimitations

We discuss the study design trade\-offs and the threats to validity in the methodology section, as recommended byrobillard\_communicating\_2024\. The current section covers the remaining limitations specific to the study objectives\.

The study is centered on offline data analysis at European XFEL\. The agenticAIsystem, including all customization options, is tailored to this task and the facility infrastructure\. Transferability to other tasks at European XFEL and other scientific facilities is uncertain\. We also expect facility users to further tailor the AGENTS\.md file to their individual needs\. Offline data analysis encompasses a wide range of use cases, and users possess varying levels of expertise\. Hence, users have to adjust the agent to their needs, including the level of computational optimization required for their use case\. Nevertheless, practitioners at and outside of European XFEL can compare the analyzed challenges and system requirements with those in their environment and adapt our system design recommendations and project artifacts accordingly\.

Only European XFEL staff tested the prototypes, not actual users\. We chose staff members as proxies for real facility users due to organizational restrictions\. The selected staff members had experience with the data analysis process, but to a varying degree\. This reflects the data analysis skills of actual users, ranging from unfamiliar to regular visitors\. For example, the user study included sessions with two PhD students who were the closest proxy for novice facility users\. The students work on specific research projects requiring beamtime, contribute to beamtime preparation, and are typically responsible for the associated data analysis\.

The two managers we interviewed in the initial phase of the study became co\-authors of this paper because they supported the design of the subsequent research activities\. Both also took part in the user study as participants because they were not actively involved in the prototype development\. During the user study, they focused on their roles as managers to provide honest feedback about the prototype\.

We have not tested production\-grade aspects of the prototype in detail\. For example, a malicious user could use the system to flood theHPCcluster with requests\. In practice, however, we consider this security aspect as a minor threat with respect to the proposed system architecture\. The agent has the same permissions and restrictions as the user who configured the system\. Anything the agent can do, a user could do as well\. What we have to test further, however, are safety aspects to prevent users from unintentionally causing harm through an agent acting autonomously\. A possible solution is a sandbox that encloses the agent and grants access only to specific server directories and capabilities\.

The tool comparison is not a benchmark study, and we did not evaluate the prototype during a long\-term deployment\. Selected system components, such as Kilo Code and Grounded Docs, may not be the best options for our use case\. However, they were good enough as they fulfilled the previously established requirements\. Practitioners of other organizations can choose a different system configuration\. However, as many agentic tools andMCPservers are interoperable, they can still reuse our design recommendations\. A long\-term deployment of the system was outside the scope of this exploratory study and is reserved for future work\.

## VRelated Work

Recent work onAIsupport forscientific data analysissuggests that assistants should combine planning and execution steps, align with scientific workflows, and adapt to a user’s expertise\.gu\_how\_2024show that effective scientificAIassistants combine coding with planning support, such as providing suggestions, alternatives, and rationale\. In their study, they find that explanations of an assistant can build trust among scientists but become distracting when they are too verbose, pointing to the value of concise and adaptable responses\.mcnutt\_design\_2023argue that assistants for computational notebooks can increase productivity while supporting the verification and reproducibility demands of scientists\. An assistant should operate across multimodal notebook context, support selective code execution, offer domain\-specific linters, and have accessible control features\.ramasamy\_ai\_2025find that code recommendations of workflow\-aware assistants can have high acceptance, especially for predictive scientific workflows\.cristea\_jelai\_2025demonstrate how context\-aware tutoring can be embedded directly in notebooks\. Together, these studies suggest that effective scientific assistants are tightly integrated into the analysis environment to provide grounded knowledge\.

Research onAIassistants for variouscoding tasksshows that they can accelerate routine programming tasks, but their usefulness declines when project context, external dependencies, or domain\-specific constraints matter\. In a previous study, we found that predefined prompts can produce more consistent and readable code documentation than ad\-hoc prompts, but that experienced developers like the interactiveness of ad\-hoc prompting\[kruse\_can\_2024\]\.corso\_generating\_2024show that code assistants often fail or hallucinate when dependencies and broader project context become important\.nettur\_role\_2025summarize that coding assistants provide productivity gains but also introduce security, reproducibility, and intellectual\-property risks\. Based on that,murali\_ai\-assisted\_2024demonstrate that domain\-adapted assistants can be adopted at scale in production when they are integrated into the environment with safeguards\. Overall, the literature indicates that practitioners must evaluate code assistants for context awareness, validation support, and safe integration into development workflows\.

Further work on therisks ofAIsupportemphasizes human control, traceability, and verifiability as mitigation strategies\.rahe\_how\_2025show that inexperienced developers can become overreliant on generated code and stop evaluating it critically\.tang\_risks\_2025argue for a safety\-first approach with constrained autonomy, traceability, and expert oversight\.kazemitabaar\_improving\_2024find that providing an editable, data\-grounded implementation plan improves scientists’ ability to verify assistant behavior\. In an earlier study, we derived ethical requirements that call for strongerAIdocumentation and tooling to operationalize and validate risk mitigation\[puhlfurs\_model\_2025\]\. These studies point to the need for transparent, bounded, and reviewable AI support in scientific settings\.

Our study builds on these strands by bringing them together in a facility\-specificDSRproject\. Instead of examining notebook assistance, code generation, or risk mitigation in isolation, we study and report how an agentic system can support offline data analysis at European XFEL for knowledge retrieval and code generation while working on a realHPCcluster\. This lets us translate prior findings into concrete requirements and design recommendations for a maintainable, human\-controlled system that fits a specialized infrastructure\.

## VIConclusion

This work addresses the problem of offline data analysis at large scientific facilities like European XFEL\. This includes complex data, fragmented knowledge, and heterogeneous user expertise, which makes effective support hard to scale\. Using aDSRapproach, we iteratively analyzed knowledge challenges, derived software requirements, built two prototypes, and evaluated them with facility experts\.

We identified 13 challenges such as users’ unfamiliarity with the infrastructure, software, and data analysis procedures, as well as the overall time required for staff members to provide support\. The key challenge for users is the need to synthesize scattered documentation, informal support, and data analysis expertise into a project\-specific data analysis workflow\.

We analyzed 21 requirements showing that a usefulAIagent must be integrated into theHPCenvironment, grounded in retrievable knowledge, able to generate and test source code, and constrained by human oversight for sensitive actions\. A critical requirement is that system components must remain replaceable and low\-maintenance so the system can adapt to user needs and the evolvingAIlandscape\.

Our user study suggests 14 design recommendations for an agenticAIsystem\. We showed that off\-the\-shelf agent tools are more practical than a custom\-built system, but only when they can be configured to match user workflows and safety constraints\. In practice, that means documentation must be agent\-ready, approvals must be configured deliberately, and behavioral control should rely on programmable hooks rather than on natural\-language instructions alone\.

Future research should test these recommendations in longer deployments and in additional scientific workflows to determine which parts generalize beyond offline data analysis at European XFEL\. In particular, programmable hooks that could ensure consistent agent behavior deserve further study\. Scientific facilities will support such agentic systems in their critical environments only if reliable guardrails are in place\.

Overall, the study shows that an agenticAIsystem can help move scientists from overload to insights by connecting knowledge retrieval, code generation, and theHPCenvironment into an integrated, modular system\. This promise depends on careful grounding, explicit safeguards, and continuous adaptation\. Without these constraints, the same system would amplify noise rather than reduce it\.

## Acknowledgment

We thank all participants for their insights, in particular the Data Analysis group of European XFEL\. We acknowledge DASHH, Data Science in Hamburg \- Helmholtz Graduate School for the Structure of Matter, for financial support\.

## References