DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis

# Deploying Human–LLM Analytics forExploratory Structured Data Analysis
Source: [https://arxiv.org/html/2610.02679](https://arxiv.org/html/2610.02679)
## DataWeave: Deploying Human–LLM Analytics for Exploratory Structured Data Analysis

Harith LaxmanVitaliy ShkremetkoEunice SonShambhavi VermaBrian O’LearyVenketesh SubramonySylvain NazefJacquelyn EliasRon CoddingtonChris ContakesMichael RileyNaren Ramakrishnan

###### Abstract

Data journalism, the practice of using data analysis to surface newsworthy stories, depends increasingly on the ability of reporters and investigative journalists to uncover trends, disparities, and accountability narratives\. In practice, exploring large structured datasets remains slow and brittle: journalists must navigate hundreds of variables across many datasets over years, understand data coding conventions, and write non\-trivial analysis code while hypotheses evolve\. Although LLMs are often touted as “ask in English, get SQL/answers,” real newsroom workflows expose recurring failures, e\.g\., schema mismatches and drift, misread domain semantics and units, and silent assumptions\. We present DataWeave, a system that addresses these needs by combining conversational interaction, schema grounding, analytical planning, and executable query generation to support exploratory analysis over structured data\. Rather than treating LLMs as autonomous answer engines, DataWeave frames them as interactive partners whose outputs can be inspected, corrected, and steered as hypotheses shift\. We present a case study with professional journalists using our system to analyze the U\.S\. Department of Education’s Integrated Postsecondary Education Data System \(IPEDS\), a high\-stakes public dataset with substantial domain semantics and frequent schema updates\. We also report how deployment experience and iterative refinement shaped the current DataWeave architecture and its analytical workflow\. Our findings distill design principles and deployment lessons for trustworthy human–LLM collaboration in structured data analysis\.

1Virginia Tech, VA, USA\.2The Chronicle of Higher Education, Washington, DC, USA

raquib@vt\.edu, naren@cs\.vt\.edu

## 1Introduction

Large\-scale structured datasets play a central role in decision making in education\([Hanushek and Raymond 2005](https://arxiv.org/html/2610.02679#bib.bib54);[Dynarski et al\. 2015](https://arxiv.org/html/2610.02679#bib.bib55)\), public policy\([Einav and Levin 2014](https://arxiv.org/html/2610.02679#bib.bib51);[Jarmin and O’Hara 2016](https://arxiv.org/html/2610.02679#bib.bib52)\), and journalism\([Kasica et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib15);[Hao et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib20);[Hamilton and Turner 2009](https://arxiv.org/html/2610.02679#bib.bib53)\)\. Despite their importance, these datasets remain difficult to explore in practice\([Milo and Somech 2020](https://arxiv.org/html/2610.02679#bib.bib18);[Kandel et al\. 2012](https://arxiv.org/html/2610.02679#bib.bib31);[Liu et al\. 2019](https://arxiv.org/html/2610.02679#bib.bib32)\)\. Analysts must navigate hundreds of variables spread across multiple tables, interpret cryptic schema documentation, and write non\-trivial code as questions evolve\([Han et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib16);[Chai et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib27)\)\. This friction disproportionately affects domain experts, such as journalists, who possess deep contextual knowledge but may lack the technical expertise or time for low\-level data wrangling\([Kandel et al\. 2011](https://arxiv.org/html/2610.02679#bib.bib25);[Chattopadhyay et al\. 2020](https://arxiv.org/html/2610.02679#bib.bib30)\)\.

Recent advances in large language models \(LLMs\) have renewed interest in conversational interfaces for data analysis\([Barbon Junior et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib17);[Drosos et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib23);[Chandel et al\. 2022](https://arxiv.org/html/2610.02679#bib.bib28);[Lai et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib29)\)\. Many existing systems cast structured data reasoning as a question–answering task, where users issue a natural language query and the model returns a summarized result\([Gomez\-Vazquez et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib26);[Chopra et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib38)\)\. Other approaches pursue fully autonomous analytical agents that plan, execute, and refine analyses without user intervention\([Hu et al\. 2025](https://arxiv.org/html/2610.02679#bib.bib24)\)\. While promising in controlled settings, these paradigms are poorly aligned with real\-world exploratory workflows\([Xie et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib22);[Wang et al\. 2018](https://arxiv.org/html/2610.02679#bib.bib56);[Feng et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib59);[Ferdowsi et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib60);[Narechania et al\. 2021](https://arxiv.org/html/2610.02679#bib.bib61)\)\. In practice, analysts rarely know the*right question*in advance\([Gu et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib19);[Jiang et al\. 2022](https://arxiv.org/html/2610.02679#bib.bib33);[Xu et al\. 2022](https://arxiv.org/html/2610.02679#bib.bib34)\), and fully autonomous execution obscures intermediate assumptions, making it difficult to trust or correct results\([Khurana et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib13);[Ning et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib14);[Li et al\. 2024a](https://arxiv.org/html/2610.02679#bib.bib3);[Hu et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib21);[Olausson et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib39)\)\.

![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/conceptual_arch.png)Figure 1:Illustration of the DataWeave interaction workflow\. A user poses a natural\-language query over higher\-education data, from which DataWeave constructs inspectable analytical artifacts including analytical assumptions, executable queries, and results \(tables or visualizations\)\. These artifacts are surfaced to the user to support verification, correction, and iterative refinement\.Our work is motivated by a year\-long collaboration with education journalists analyzing the U\.S\. Department of Education’s Integrated Postsecondary Education Data System \(IPEDS\)\. IPEDS is a large, heterogeneous collection of tabular datasets covering enrollment, demographics, finances, and outcomes across thousands of U\.S\. educational institutions\. Although publicly available, IPEDS poses significant challenges: schemas vary across years and tables, column semantics are highly domain\-specific, and missing or inconsistently reported values are common\. Journalists working with IPEDS emphasized the need not for automated answers, but for tools that support*iterative sensemaking*, including the ability to explore trends, test hypotheses, inspect intermediate results, and correct errors as understanding evolves\.

Through early prototypes, we observed that fully autonomous LLM\-driven analytics often failed in subtle but consequential ways\. Examples included assuming the presence of nonexistent columns, misinterpreting table semantics, or silently producing misleading aggregates\. Crucially, these errors were frequently easy for domain experts to detect and fix once exposed\. This observation led to a key insight: for structured data analysis,*human guidance is not a limitation, but a strength*\. Small, well\-timed user interventions can substantially improve analytical correctness, interpretability, and trust\.

Based on these findings, we presentDataWeave, an interactive, human\-in\-the\-loop system for exploratory analysis of structured and numerical data using LLMs\. DataWeave combines intent interpretation, schema grounding, analytical planning, query formulation, and bounded execution within an agent\-mediated conversational workflow, while users retain control over analytical goals and interpretation\. Users articulate open\-ended analytical goals, while the system generates schema\-grounded executable queries and preserves intermediate analytical traces for inspection and refinement\. DataWeave supports cross\-table joins, longitudinal comparisons, and subgroup analyses common in data journalism workflows\. Figure[1](https://arxiv.org/html/2610.02679#S1.F1)shows the interactive analysis workflow\.

The current architecture emerged through continued deployment and refinement, where experience with an earlier stage\-wise implementation informed the final organization of these analytical functions\. Our contributions are:

1. 1\.We introduceDataWeave, a deployed, human\-in\-the\-loop analytics system for use by professional journalists\. As part of this study, we identify recurring failure modes in LLM\-based structured data analysis, e\.g\., schema drift, silent aggregation errors, and misaligned temporal assumptions, and show how exposing intermediate artifacts makes them readily correctable\.
2. 2\.We distill a set of transferable design principles for human–LLM collaboration in high\-stakes analytical workflows, including clarification before assumption, objective\-conditioned schema grounding, inspectable executable queries, bounded recovery, and guided agent delegation\.
3. 3\.We report deployment experience from professional newsroom use and show how iterative refinement of the system led to the current agent\-mediated architecture while preserving its core analytical workflow\.
4. 4\.Through deployment analysis and a retrospective IPEDS case study, we demonstrate how these design choices support iterative sensemaking while preserving editorial control and accountability\.

![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/chron_iaai.png)Figure 2:DataWeave architecture\. User objectives are interpreted and clarified, grounded through objective\-conditioned schema retrieval, translated into analytical plans and read\-only SQL, and executed with bounded recovery\. Queries, results, and errors are preserved as analytical traces to support transparency and reproducibility\.Table 1:Condensed view of our design principles, what they mean, and how DataWeave realizes them\.
## 2Related Work

Recent research has examined how LLMs can assist with structured data analysis via natural language interfaces, program synthesis, and visualization support\([Wu et al\. 2024b](https://arxiv.org/html/2610.02679#bib.bib46);[Liu et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib57);[Liu et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib58)\)\. OpenTab\([Kong et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib1)\), for example, studies open\-domain table reasoning where models must first identify relevant tables and then generate SQL and answers\. InsightBench\([Sahu et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib2)\)proposes a benchmark for evaluating agents that autonomously generate multi\-step insights from curated datasets\. Other systems emphasize interaction and visualization\([Shen et al\. 2022](https://arxiv.org/html/2610.02679#bib.bib40)\), such as LIDA\([Dibia 2023](https://arxiv.org/html/2610.02679#bib.bib5)\)and Chat2VIS\([Maddigan and Susnjak 2023](https://arxiv.org/html/2610.02679#bib.bib6);[Shen et al\. 2022](https://arxiv.org/html/2610.02679#bib.bib40)\), which convert user requests into charts and lightweight exploratory analyses\([Tian et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib50)\)\. Mixed\-initiative tools like Data Formulator and Data Formulator 2\([Wang et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib7);[Wang et al\. 2025](https://arxiv.org/html/2610.02679#bib.bib8)\)support iterative specification of data transformations and visual encodings\. Code\-oriented assistants including GPT\-4\([Achiam et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib12);[Baktash and Dawodi 2023](https://arxiv.org/html/2610.02679#bib.bib41);[Rodrigues et al\. 2025](https://arxiv.org/html/2610.02679#bib.bib42);[Stribling et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib43)\)as a Data Analyst\([Cheng et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib9);[Huang et al\. 2024](https://arxiv.org/html/2610.02679#bib.bib44);[Li et al\. 2024b](https://arxiv.org/html/2610.02679#bib.bib45)\)and Data\-Copilot\([Zhang et al\. 2023](https://arxiv.org/html/2610.02679#bib.bib10)\)help users generate and execute analysis code, while agent frameworks such as AutoGen\([Wu et al\. 2024a](https://arxiv.org/html/2610.02679#bib.bib11)\)investigate multi\-agent coordination for complex tasks\([Dorri et al\. 2018](https://arxiv.org/html/2610.02679#bib.bib47)\)\.

A closely related line of work is Text\-to\-SQL, which studies how to translate natural language questions into executable SQL over relational databases\([Luo et al\. 2025](https://arxiv.org/html/2610.02679#bib.bib49)\)\. Recent advances have improved schema linking\([Wang et al\. 2020](https://arxiv.org/html/2610.02679#bib.bib48)\), prompt design, and execution\-guided generation\([Liu et al\. 2025](https://arxiv.org/html/2610.02679#bib.bib4);[Li et al\. 2024a](https://arxiv.org/html/2610.02679#bib.bib3);[Yu et al\. 2019](https://arxiv.org/html/2610.02679#bib.bib35);[Xie et al\. 2025](https://arxiv.org/html/2610.02679#bib.bib36);[Scholak et al\. 2021](https://arxiv.org/html/2610.02679#bib.bib37)\)\. These systems typically assume a well\-posed question and a relatively stable schema\([Li et al\. 2024a](https://arxiv.org/html/2610.02679#bib.bib3)\), and aim to produce a correct query for execution\.

DataWeave overlaps with prior work in using LLMs to generate executable analytical artifacts, but targets high\-stakes exploratory analysis over heterogeneous, evolving public datasets\. In this setting, questions are underspecified, schemas drift across releases, and analysts refine goals iteratively\. Accordingly, DataWeave emphasizes objective\-conditioned schema grounding, analytical planning, constrained query execution, and persistent traces that record assumptions, errors, and revisions\. Rather than optimizing for autonomous insight generation or rapid visualization, DataWeave is designed for collaborative investigative workflows where human oversight is central and intermediate steps remain visible and contestable\.

## 3System Architecture

At a high level, DataWeave combines LLM\-based intent interpretation and analytical planning with schema\-grounded query formulation and deterministic execution over structured IPEDS data\. User interactions are coordinated through an agent\-mediated workflow comprising intent interpretation and clarification, schema grounding, analytical planning, query formulation, execution and recovery, and answer synthesis with persistent analytical traces\. Figure[2](https://arxiv.org/html/2610.02679#S1.F2)shows the core system modules and their roles\.

### 3\.1Front\-End and Session Management

DataWeave provides a conversational web interface that maintains a persistent analysis session per authenticated user\. The interface shows the conversation, clarification requests, analytical activity, and returned results, with tools to browse prior logs and manage sessions\. Persistent state maintains conversation history and the analytical context required to support follow\-up questions, while per\-user storage preserves prior turns, outputs, errors, and user\-approved preferences\. Authentication and per\-user storage support isolated, long\-running analytical workflows\.

### 3\.2Data and Metadata Representation

DataWeave operates over multiple table families spanning time periods\. Each IPEDS collection year is represented through year\-specific relational tables together with metadata catalogs describing tables, variables, data types, and categorical value sets\. Upon ingestion, table and column names are normalized to reduce brittleness in query formulation\.

Rather than assuming a fixed global schema, DataWeave treats schema information as year\-specific metadata that may vary across releases\. This representation allows downstream components to reason about table and variable selection, categorical value semantics, and temporal alignment as the schema changes over time\. Instead of placing the complete schema in the model context, DataWeave retrieves relevant table, variable, and value\-set metadata on demand and uses this information as structured grounding for subsequent analysis\.

### 3\.3Intent Interpretation and Clarification

Given a user turn, DataWeave interprets the analytical objective and determines whether sufficient information is available to proceed\. If the objective contains an ambiguous abbreviation, missing temporal scope, or other information required for analysis, the system requests a concise clarification before querying the data\. The clarified objective then provides the basis for schema grounding and subsequent analysis\.

### 3\.4Schema Grounding

Given the clarified analytical objective, DataWeave retrieves relevant information from the year\-specific metadata catalog\. The system first identifies candidate tables and then retrieves descriptions of relevant variables and, when needed, the allowed values and meanings of categorical codes\. This objective\-conditioned retrieval provides a compact grounded schema context without requiring the complete database schema to be included in the model prompt\.

### 3\.5Analytical Planning

Given the analytical objective and grounded schema context, DataWeave forms an analytical plan specifying the tables, variables, filters, temporal scope, and analytical operations required to answer the question\. Planning is grounded in retrieved schema and semantic information, including the meanings of coded variables, rather than relying on model memory\. The resulting analytical specification constrains subsequent query formulation\.

### 3\.6Query Formulation

Based on the analytical plan, DataWeave formulates one or more read\-only SQL queries that retrieve, filter, join, and aggregate the required data\. Query formulation is constrained by the grounded schema and domain\-specific rules, including year\-specific table structure, variable semantics, and categorical codes\. Table and variable names are therefore derived from retrieved metadata rather than assumed by the model\.

### 3\.7Execution and Recovery

Generated queries are executed against a read\-only data store, with execution restricted to retrieval operations and bounded result sets\. Execution can produce either concrete query results or explicit errors\. When a query fails, the execution feedback is returned to the analytical workflow, allowing the query to be reformulated using the original objective, grounded schema context, and observed error\. Recovery remains bounded rather than allowing uncontrolled execution\. Queries, returned results, and errors are retained as part of the analytical trace\.

### 3\.8Answer Synthesis and Trace Persistence

DataWeave synthesizes the final response from the results returned through successful query execution rather than from unsupported model assumptions\. The response retains relevant provenance, such as the collection year and source tables, while the underlying interaction is preserved as an analytical trace containing the user objective, tool interactions, queries, returned results, and execution errors\.

These traces are persisted across sessions, enabling users to inspect prior analytical steps, continue an investigation through follow\-up questions, and resume earlier exploratory workflows\. Persistent conversational state also allows clarification and other user interventions to remain part of the analytical process\.

### 3\.9Design Principles behind DataWeave

The process of deploying, evaluating, and iteratively refining DataWeave has led us to articulate key design principles as shown in Table[1](https://arxiv.org/html/2610.02679#S1.T1)\. We highlight them here as potential takeaways for the design of reliable human\-LLM analytics systems for complex, heterogeneous structured datasets\.

## 4Deployment Experience and System Evolution

Table 2:Summary of deployment usage\.Table 3:Alignment between planner\-inferred columns and columns actually used in execution \(deployment logs\)\. Higher overlap indicates stronger schema grounding\.DataWeave was deployed for use by professional journalists and data analysts at*The Chronicle of Higher Education*\. We collected interaction logs to characterize how the system was used in practice and how its design evolved through deployment\. The initial deployed version used explicit stage\-wise orchestration, while subsequent refinements led to the agent\-mediated architecture described in Section[3](https://arxiv.org/html/2610.02679#S3)\.

### 4\.1Deployment and Iterative Use

Deployment logs contain 298 analysis sessions comprising 797 analytical steps from 18 users \(Table[2](https://arxiv.org/html/2610.02679#S4.T2)\)\. Usage extended beyond isolated interactions: the median user completed eight sessions, and 12 of the 18 users completed at least five sessions\. Sessions frequently extended beyond a single query and included follow\-up questions and refinements of earlier analyses\.

To characterize these workflows, each analytical step was assigned one*Dominant Analytical Intent*\(DAI\), describing its primary purpose, and one or more*Analytical Dimension Tags*\(ADT\), describing secondary properties such as comparison or demographic focus \(Figure[3](https://arxiv.org/html/2610.02679#S4.F3)\)\. A substantial fraction of transitions between steps refined a previous objective, indicating that DataWeave was commonly used as an iterative analytical workspace rather than a single\-turn question\-answering system\.

### 4\.2Schema Grounding and Planning Behavior

Deployment logs also allow us to examine how the system selected and used the underlying IPEDS schema\. Core identifiers such asunitidandinstnm, together with frequently used enrollment variables such asefalevelandeftotlt, appeared most often in executed analyses \(Figure[4](https://arxiv.org/html/2610.02679#S4.F4)\)\.

The analytical planner generally identified a broader set of candidate columns than were ultimately used during execution \(Table[3](https://arxiv.org/html/2610.02679#S4.T3)and Figure[5](https://arxiv.org/html/2610.02679#S4.F5)\)\. This pattern suggests conservative schema coverage followed by task\-specific selection\. Column usage also tended to increase in later steps of longer sessions \(Figure[6](https://arxiv.org/html/2610.02679#S4.F6)\), consistent with investigations becoming more detailed as users refined their questions\.

### 4\.3From Staged to Agent\-Mediated Orchestration

The initial deployed version organized intent handling, schema grounding, planning, code generation, and execution as explicit application\-defined stages\. As DataWeave evolved, these core functions were retained, but their coordination shifted toward ReAct\-style agents that dynamically sequence schema discovery, planning, query formulation, and execution feedback\.

The final architecture therefore preserves the original analytical structure while allowing the execution path to adapt more directly to the task\.

![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/primary.png)\(a\)Primary intents \(DAI\)\.
![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/tags.png)\(b\)Analytical dimension tags \(ADT\)\.

Figure 3:Distribution of analytical patterns across real\-world analytical steps\. \(a\) Each step is assigned one dominant analytical intent \(Data quality validation withn=1n=1not shown\)\. \(b\) Steps may also receive multiple analytical dimension tags \(ADT\) capturing additional analytical properties\.![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/top_used_columns.png)Figure 4:Most frequently used columns across real\-world analytical steps\. Core identifiers \(e\.g\.,unitid,instnm\) and enrollment\-related variables dominate usage, reflecting consistent grounding in entity resolution and enrollment analysis\.![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/planned_vs_used_hist.png)Figure 5:Distribution of planner\-inferred versus executed columns per analytical step\. The planner typically proposes a broader column set, while execution uses a smaller subset\. This pattern indicates conservative schema grounding followed by selective use during execution\.![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/columns_per_step_progression.png)Figure 6:Column footprint across step progression within sessions\. Thin lines show individual sessions and the bold line shows the median\. Later steps often involve broader column usage, reflecting refinement and deeper investigation\.![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/fig1_headline_metrics.png)\(a\)Headline evaluation metrics at the 1% error tolerance\.
![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/fig2_accuracy_by_tolerance.png)\(b\)Accuracy across relative\-error tolerances\.

Figure 7:Answer correctness across the evaluated DataWeave configurations\. The earlier prototype baseline is evaluated on a 100\-question subset covering the five most recent years, while the agentic configurations are evaluated on the full 400\-question benchmark\. Error bars show Wilson 95% confidence intervals\.![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/fig5_tool_calls.png)\(a\)Distribution of tool calls per question\.
![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/fig3_latency.png)\(b\)Latency per question\.
![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/fig4_cost.png)\(c\)Cost per question\.

Figure 8:Operational behavior of the evaluated configurations\. Tool\-call statistics are available for the agentic configurations; the earlier prototype harness does not record tool calls\. Latency is reported for all evaluated runs\.![Refer to caption](https://arxiv.org/html/2610.02679v1/figures/use_case.png)Figure 9:Retrospective case study aligning DataWeave analyses with a 2021*Chronicle of Higher Education*article on graduate enrollment trends\. The figure shows selected steps in a reconstructed analytical workflow, where DataWeave queries reproduce reported statistics\. Arrows link computed outputs to corresponding claims and tables in the article, illustrating how conversational analysis can support journalistic verification and story development\.
### 4\.4Operational Implications

Agent\-mediated orchestration increased analytical flexibility, but also made schema exploration and computational cost more dependent on the agent’s reasoning path\. This reinforced the need for structured schema access, explicit planning, constrained execution, and persistent traces around the agent workflow\. The evolution also showed that analytical flexibility and integration compatibility are separate deployment considerations\.

## 5Evaluation

The deployment analysis in the previous section characterizes how DataWeave is used in open\-ended newsroom workflows\. Here, we evaluate the system on structured questions with published reference answers and examine both answer quality and operational behavior\.

We structure the evaluation around three questions:

- •RQ1: Answer Correctness\.How accurately does DataWeave answer structured IPEDS questions with known reference values?
- •RQ2: Operational Behavior\.How do the evaluated system configurations differ in tool use, latency, and computational cost?
- •RQ3: Investigative Utility\.Can DataWeave reproduce and support analytical trajectories underlying real journalistic reporting?

RQ1–RQ2 use a benchmark derived from published IPEDS Trend Generator questions, while RQ3 reconstructs the analysis behind a previously published article\.

### 5\.1Evaluation Setup

We evaluate DataWeave on 400 questions derived from 45 published IPEDS Trend Generator questions spanning 12 subject areas and 20 collection years \(2004–05 through 2023–24\)\. Questions include direct totals, institution\-filtered totals, and categorical breakdowns, and are sampled with a fixed seed across years and question types\.

Each template is rewritten once into natural language while preserving the requested measure, year, and analytical dimension\. Rewrites are programmatically checked, with the original template used as fallback when needed\.

Because published Trend Generator values assume active, Title IV\-participating institutions in the 50 states and the District of Columbia, this scope is supplied through an evaluation\-only prompt suffix\. The deployed system prompt and tools are otherwise unchanged\.

The two agentic configurations are evaluated on all 400 questions\. The earlier prototype baseline is evaluated on 100 questions from the five most recent years because its execution design requires the relevant year\-specific tables to be loaded into memory\.

Accuracy is measured within 1% relative error, or within 0\.5 percentage points for percentage\-valued answers\. We also report answer rate, completion rate, latency, tool calls, and cost\.

### 5\.2RQ1: Answer Correctness

As shown in Figure[7\(a\)](https://arxiv.org/html/2610.02679#S4.F7.sf1), the baseline achieves 7\.0% accuracy, compared with 70\.0% for Agentic Exploration and 72\.8% for Agentic RAG\. Both agentic configurations answer nearly all questions and complete all runs\.

Figure[7\(b\)](https://arxiv.org/html/2610.02679#S4.F7.sf2)shows that this gap remains across broader error tolerances\. The two agentic configurations perform similarly, suggesting that retrieval\-based grounding does not materially change answer correctness on this benchmark\.

### 5\.3RQ2: Operational Behavior

We next examine the execution behavior of the agentic configurations\. Agentic Exploration and Agentic RAG use a similar number of tool calls, with mean values of 8\.80 and 9\.02, respectively, and a median of eight calls for both \(Figure[8](https://arxiv.org/html/2610.02679#S4.F8)\)\. This suggests that retrieval\-based grounding does not substantially change the number of analytical interactions required per question\.

Latency is also similar across the two configurations, with median values of 18\.5 s for Agentic Exploration and 18\.0 s for Agentic RAG\. The earlier prototype shows a higher median latency of 28\.4 s on its evaluated subset\.

Operational cost differs more substantially\. Mean cost per question is $0\.0601 for the baseline, $0\.0330 for Agentic Exploration, and $0\.0065 for Agentic RAG\. Costs reflect the accounting available from each evaluation backend and are therefore interpreted as operational rather than strictly normalized costs\.

### 5\.4RQ3: Investigative Utility in Story Construction

To answer RQ3, we evaluate whether DataWeave can support investigative workflows comparable to real journalistic practice\. We do so through a retrospective case study based on a previously published 2021*Chronicle of Higher Education*article on graduate enrollment trends\. The article predates DataWeave and was produced independently by journalists\. Our goal is not to generate the article, but to reconstruct the analytical workflow underlying its claims and assess whether DataWeave can reproduce the reported statistics and support similar lines of inquiry\.

As shown in Figure[9](https://arxiv.org/html/2610.02679#S4.F9), story construction is typically sequential: reporters detect a signal, segment it by subgroup, validate sector\-level claims, and identify institution\-level examples\. In this example, beginning with baseline U\.S\. graduate enrollment trends, the story narrows down into particular sectors, surfacing outliers as story angles, followed by subgroup validation\.

This retrospective reconstruction shows that DataWeave can reproduce key reported statistics and support the sequential analytical moves observed in real reporting\. The system functions as a query\-to\-story scaffold rather than a single\-turn answer engine\. By enabling signal detection, segmentation, validation, and outlier discovery with inspectable computations, DataWeave supports investigative sensemaking and verification\.

## 6Lessons Learned

Our deployment of DataWeave with professional journalists yielded the following lessons for human–LLM structured data analysis\.

Collaboration Better Fits Investigative Workflows\.Users treated DataWeave as a collaborative partner rather than an autonomous analyst\. The system supported analysis and query generation, while journalists retained control over interpretation and decision making\. This suggests that inspectable and steerable workflows are better suited to exploratory, high\-stakes analysis than end\-to\-end autonomy\.

Transparency supports Trust and Accountability\.Journalists regularly inspected generated queries and intermediate outputs before accepting results, reflecting professional norms around verification and defensibility\. By making analytical steps explicit, DataWeave supported verification practices central to data journalism\. Similar demands arise in policy, public health, and education, where analyses must be explainable, revisitable, and defensible\.

Agent Delegation Shifts Analytical Complexity\.Agent\-mediated orchestration increases flexibility, but shifts more complexity into the model’s reasoning path\. This can produce broader and less predictable schema exploration, reinforcing the need for structured grounding, constrained execution, and persistent traces\.

Deployment Fit Extends Beyond Analytical Capability\.A system may provide the required analytical capability but still differ from how an organization expects to integrate it into existing workflows\. Our deployment experience showed that reasoning flexibility, efficiency, and integration compatibility should be treated as separate design considerations\.

## 7Conclusion

Through deployment with professional education journalists, we showed that conversational, query\-centric workflows can support iterative sensemaking while preserving transparency and user control\. Deployment experience shaped DataWeave’s current architecture, combining schema grounding, analytical planning, agent\-mediated coordination, and constrained execution\. The results further show that these design choices can substantially improve correctness while maintaining practical latency and cost\. More broadly, reliable LLM analytics should balance agent flexibility with structured grounding, human oversight, and deployment compatibility\. Future work will extend DataWeave to additional domains and explore further forms of guided autonomy while preserving human control\.

## References

- Achiamet al\.\(2023\)J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Baktash and Dawodi \(2023\)J\. A\. Baktash and M\. DawodiGpt\-4: a review on advancements and opportunities in natural language processing\.arXiv preprint arXiv:2305\.03195\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Barbon Junioret al\.\(2024\)S\. Barbon Junior, P\. Ceravolo, S\. Groppe, M\. Jarrar, S\. Maghool, F\. Sèdes, S\. Sahri, and M\. Van KeulenAre large language models the new interface for data pipelines?\.InProceedings of the International Workshop on Big Data in Emergent Distributed Environments,pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Chaiet al\.\(2023\)C\. Chai, N\. Tang, J\. Fan, and Y\. LuoDemystifying artificial intelligence for data preparation\.InCompanion of the 2023 International Conference on Management of Data,pp\. 13–20\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Chandelet al\.\(2022\)S\. Chandel, C\. B\. Clement, G\. Serrato, and N\. SundaresanTraining and evaluating a jupyter notebook data science assistant\.arXiv preprint arXiv:2201\.12901\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Chattopadhyayet al\.\(2020\)S\. Chattopadhyay, I\. Prasad, A\. Z\. Henley, A\. Sarma, and T\. BarikWhat’s wrong with computational notebooks? pain points, needs, and design opportunities\.InProceedings of the 2020 CHI conference on human factors in computing systems,pp\. 1–12\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Chenget al\.\(2023\)L\. Cheng, X\. Li, and L\. BingIs gpt\-4 a good data analyst?\.arXiv preprint arXiv:2305\.15038\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Chopraet al\.\(2023\)B\. Chopra, A\. Singha, A\. Fariha, S\. Gulwani, C\. Parnin, A\. Tiwari, and A\. Z\. HenleyConversational challenges in ai\-powered data science: obstacles, needs, and design opportunities\.arXiv preprint arXiv:2310\.16164\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Dibia \(2023\)V\. DibiaLIDA: a tool for automatic generation of grammar\-agnostic visualizations and infographics using large language models\.arXiv preprint arXiv:2303\.02927\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Dorriet al\.\(2018\)A\. Dorri, S\. S\. Kanhere, and R\. JurdakMulti\-agent systems: a survey\.IEEE Access6,pp\. 28573–28593\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Drososet al\.\(2024\)I\. Drosos, A\. Sarkar, X\. Xu, C\. Negreanu, S\. Rintel, and L\. Tankelevitch" It’s like a rubber duck that talks back": understanding generative ai\-assisted data analysis workflows through a participatory prompting study\.InProceedings of the 3rd Annual Meeting of the Symposium on Human\-Computer Interaction for Work,pp\. 1–21\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Dynarskiet al\.\(2015\)S\. M\. Dynarski, S\. W\. Hemelt, and J\. M\. HymanThe missing manual: using national student clearinghouse data to track postsecondary outcomes\.Educational Evaluation and Policy Analysis37\(1\_suppl\),pp\. 53S–79S\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Einav and Levin \(2014\)L\. Einav and J\. LevinThe data revolution and economic analysis\.Innovation policy and the economy14\(1\),pp\. 1–24\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Fenget al\.\(2023\)Y\. Feng, X\. Wang, B\. Pan, K\. K\. Wong, Y\. Ren, S\. Liu, Z\. Yan, Y\. Ma, H\. Qu, and W\. ChenXnli: explaining and diagnosing nli\-based visual data analysis\.IEEE Transactions on Visualization and Computer Graphics30\(7\),pp\. 3813–3827\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Ferdowsiet al\.\(2023\)K\. Ferdowsi, J\. Williams, I\. Drosos, A\. D\. Gordon, C\. Negreanu, N\. Polikarpova, A\. Sarkar, and B\. ZornColDeco: an end user spreadsheet inspection tool for ai\-generated code\.In2023 IEEE Symposium on Visual Languages and Human\-Centric Computing \(VL/HCC\),pp\. 82–91\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Gomez\-Vazquezet al\.\(2024\)M\. Gomez\-Vazquez, J\. Cabot, and R\. ClarisóAutomatic generation of conversational interfaces for tabular data analysis\.InProceedings of the 6th ACM Conference on Conversational User Interfaces,pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Guet al\.\(2024\)K\. Gu, R\. Shang, T\. Althoff, C\. Wang, and S\. M\. DruckerHow do analysts understand and verify ai\-assisted data analyses?\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,pp\. 1–22\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Hamilton and Turner \(2009\)J\. T\. Hamilton and F\. TurnerAccountability through algorithm\.InCenter for Advanced Study in the Behavioral Sciences Summer Workshop\. Available online: http://web\. stanford\. edu/˜\{\\\{\}\\\}fturner/Hamilton% 20Turner% 20Acc% 20by% 20Alg% 20Final\. pdf \(accessed on 5 February 2021\),Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Hanet al\.\(2023\)L\. Han, T\. Chen, G\. Demartini, M\. Indulska, and S\. SadiqA data\-driven analysis of behaviors in data curation processes\.ACM Transactions on Information Systems41\(3\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Hanushek and Raymond \(2005\)E\. A\. Hanushek and M\. E\. RaymondDoes school accountability lead to improved student performance?\.Journal of Policy Analysis and Management: The Journal of the Association for Public Policy Analysis and Management24\(2\),pp\. 297–327\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Haoet al\.\(2024\)S\. Hao, Z\. Wang, B\. Bach, and L\. PschetzDesign patterns for data\-driven news articles\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,pp\. 1–16\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Huet al\.\(2025\)M\. Hu, P\. Zhao, C\. Xu, Q\. Sun, J\. Lou, Q\. Lin, P\. Luo, and S\. RajmohanAgentgen: enhancing planning abilities for large language model based agent via environment and task generation\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 1,pp\. 496–507\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Huet al\.\(2024\)Z\. Hu, Y\. Zhang, R\. Rossi, T\. Yu, S\. Kim, and S\. PanAre large language models capable of causal reasoning for sensing data analysis?\.InProceedings of the Workshop on Edge and Mobile Foundation Models,pp\. 24–29\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Huanget al\.\(2024\)Y\. Huang, R\. Wu, J\. He, and Y\. XiangEvaluating chatgpt\-4\.0’s data analytic proficiency in epidemiological studies: a comparative analysis with sas, spss, and r\.Journal of Global Health14,pp\. 04070\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Jarmin and O’Hara \(2016\)R\. S\. Jarmin and A\. B\. O’HaraBig data and the transformation of public policy analysis\.Journal of Policy Analysis and Management35\(3\),pp\. 715–721\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Jianget al\.\(2022\)E\. Jiang, E\. Toh, A\. Molina, K\. Olson, C\. Kayacik, A\. Donsbach, C\. J\. Cai, and M\. TerryDiscovering the syntax and strategies of natural language programming with generative language models\.InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems,pp\. 1–19\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Kandelet al\.\(2011\)S\. Kandel, A\. Paepcke, J\. Hellerstein, and J\. HeerWrangler: interactive visual specification of data transformation scripts\.InProceedings of the sigchi conference on human factors in computing systems,pp\. 3363–3372\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Kandelet al\.\(2012\)S\. Kandel, A\. Paepcke, J\. M\. Hellerstein, and J\. HeerEnterprise data analysis and visualization: an interview study\.IEEE transactions on visualization and computer graphics18\(12\),pp\. 2917–2926\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Kasicaet al\.\(2023\)S\. Kasica, C\. Berret, and T\. MunznerDirty data in the newsroom: comparing data preparation in journalism and data science\.InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems,pp\. 1–18\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Khuranaet al\.\(2024\)A\. Khurana, H\. Subramonyam, and P\. K\. ChilanaWhy and when llm\-based assistants can go wrong: investigating the effectiveness of prompt\-based interactions for software help\-seeking\.InProceedings of the 29th International Conference on Intelligent User Interfaces,pp\. 288–303\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Konget al\.\(2024\)K\. Kong, J\. Zhang, Z\. Shen, B\. Srinivasan, C\. Lei, C\. Faloutsos, H\. Rangwala, and G\. KarypisOpentab: advancing large language models as open\-domain table reasoners\.arXiv preprint arXiv:2402\.14361\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Laiet al\.\(2023\)Y\. Lai, C\. Li, Y\. Wang, T\. Zhang, R\. Zhong, L\. Zettlemoyer, W\. Yih, D\. Fried, S\. Wang, and T\. YuDS\-1000: a natural and reliable benchmark for data science code generation\.InInternational Conference on Machine Learning,pp\. 18319–18345\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Liet al\.\(2024a\)B\. Li, Y\. Luo, C\. Chai, G\. Li, and N\. TangThe dawn of natural language to sql: are we fully ready?\.arXiv preprint arXiv:2406\.01265\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1),[§2](https://arxiv.org/html/2610.02679#S2.p2.1)\.
- Liet al\.\(2024b\)K\. D\. Li, A\. M\. Fernandez, R\. Schwartz, N\. Rios, M\. N\. Carlisle, G\. M\. Amend, H\. V\. Patel, and B\. N\. BreyerComparing gpt\-4 and human researchers in health care data analysis: qualitative description study\.Journal of Medical Internet Research26,pp\. e56500\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Liuet al\.\(2019\)J\. Liu, N\. Boukhelifa, and J\. R\. EaganUnderstanding the role of alternatives in data analysis practices\.IEEE transactions on visualization and computer graphics26\(1\),pp\. 66–76\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Liuet al\.\(2023\)S\. Liu, S\. Wang, T\. Chang, W\. Lin, C\. Hsiung, Y\. Hsieh, Y\. Cheng, S\. Luo, and J\. ZhangJarviX: a llm no code platform for tabular data analysis and optimization\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 622–630\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Liuet al\.\(2024\)S\. Liu, H\. Miao, Z\. Li, M\. Olson, V\. Pascucci, and P\. BremerAVA: towards autonomous visualization agents through visual perception\-driven decision\-making\.InComputer Graphics Forum,Vol\.43,pp\. e15093\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Liuet al\.\(2025\)X\. Liu, S\. Shen, B\. Li, P\. Ma, R\. Jiang, Y\. Zhang, J\. Fan, G\. Li, N\. Tang, and Y\. LuoA survey of text\-to\-sql in the era of llms: where are we, and where are we going?\.IEEE Transactions on Knowledge and Data Engineering\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p2.1)\.
- Luoet al\.\(2025\)Y\. Luo, G\. Li, J\. Fan, C\. Chai, and N\. TangNatural language to sql: state of the art and open problems\.Proceedings of the VLDB Endowment18\(12\),pp\. 5466–5471\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p2.1)\.
- Maddigan and Susnjak \(2023\)P\. Maddigan and T\. SusnjakChat2vis: generating data visualizations via natural language using chatgpt, codex and gpt\-3 large language models\.Ieee Access11,pp\. 45181–45193\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Milo and Somech \(2020\)T\. Milo and A\. SomechAutomating exploratory data analysis via machine learning: an overview\.InProceedings of the 2020 ACM SIGMOD international conference on management of data,pp\. 2617–2622\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p1.1)\.
- Narechaniaet al\.\(2021\)A\. Narechania, A\. Fourney, B\. Lee, and G\. RamosDIY: assessing the correctness of natural language to sql systems\.InProceedings of the 26th International Conference on Intelligent User Interfaces,pp\. 597–607\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Ninget al\.\(2024\)Z\. Ning, Y\. Tian, Z\. Zhang, T\. Zhang, and T\. J\. LiInsights into natural language database query errors: from attention misalignment to user handling strategies\.ACM Transactions on Interactive Intelligent Systems14\(4\),pp\. 1–32\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Olaussonet al\.\(2023\)T\. X\. Olausson, J\. P\. Inala, C\. Wang, J\. Gao, and A\. Solar\-LezamaIs self\-repair a silver bullet for code generation?\.arXiv preprint arXiv:2306\.09896\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Rodrigueset al\.\(2025\)L\. Rodrigues, C\. Xavier, N\. Costa, D\. Gasevic, and R\. F\. MelloIs gpt\-4 fair? an empirical analysis in automatic short answer grading\.Computers and Education: Artificial Intelligence,pp\. 100428\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Sahuet al\.\(2024\)G\. Sahu, A\. Puri, J\. Rodriguez, A\. Abaskohi, M\. Chegini, A\. Drouin, P\. Taslakian, V\. Zantedeschi, A\. Lacoste, D\. Vazquez,et al\.Insightbench: evaluating business analytics agents through multi\-step insight generation\.arXiv preprint arXiv:2407\.06423\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Scholaket al\.\(2021\)T\. Scholak, N\. Schucher, and D\. BahdanauPICARD: parsing incrementally for constrained auto\-regressive decoding from language models\.arXiv preprint arXiv:2109\.05093\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p2.1)\.
- Shenet al\.\(2022\)L\. Shen, E\. Shen, Y\. Luo, X\. Yang, X\. Hu, X\. Zhang, Z\. Tai, and J\. WangTowards natural language interfaces for data visualization: a survey\.IEEE transactions on visualization and computer graphics29\(6\),pp\. 3121–3144\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Striblinget al\.\(2024\)D\. Stribling, Y\. Xia, M\. K\. Amer, K\. S\. Graim, C\. J\. Mulligan, and R\. RenneThe model student: gpt\-4 performance on graduate biomedical science exams\.Scientific Reports14\(1\),pp\. 5670\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Tianet al\.\(2024\)Y\. Tian, W\. Cui, D\. Deng, X\. Yi, Y\. Yang, H\. Zhang, and Y\. WuChartgpt: leveraging llms to generate charts from abstract natural language\.IEEE Transactions on Visualization and Computer Graphics31\(3\),pp\. 1731–1745\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Wanget al\.\(2018\)A\. Y\. Wang, R\. Mitts, P\. J\. Guo, and P\. K\. ChilanaMismatch of expectations: how modern learning resources fail conversational programmers\.InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems,pp\. 1–13\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Wanget al\.\(2020\)B\. Wang, R\. Shin, X\. Liu, O\. Polozov, and M\. RichardsonRAT\-sql: relation\-aware schema encoding and linking for text\-to\-sql parsers\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 7567–7578\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p2.1)\.
- Wanget al\.\(2025\)C\. Wang, B\. Lee, S\. M\. Drucker, D\. Marshall, and J\. GaoData formulator 2: iterative creation of data visualizations, with ai transforming data along the way\.InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems,pp\. 1–17\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Wanget al\.\(2023\)C\. Wang, J\. Thompson, and B\. LeeData formulator: ai\-powered concept\-driven visualization authoring\.IEEE Transactions on Visualization and Computer Graphics30\(1\),pp\. 1128–1138\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Wuet al\.\(2024a\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.Autogen: enabling next\-gen llm applications via multi\-agent conversations\.InFirst Conference on Language Modeling,Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Wuet al\.\(2024b\)Y\. Wu, Y\. Wan, H\. Zhang, Y\. Sui, W\. Wei, W\. Zhao, G\. Xu, and H\. JinAutomated data visualization from natural language via large language models: an exploratory study\.Proceedings of the ACM on Management of Data2\(3\),pp\. 1–28\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.
- Xieet al\.\(2024\)L\. Xie, C\. Zheng, H\. Xia, H\. Qu, and C\. Zhu\-TianWaitgpt: monitoring and steering conversational llm agent in data analysis with on\-the\-fly code visualization\.InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology,pp\. 1–14\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Xieet al\.\(2025\)X\. Xie, G\. Xu, L\. Zhao, and R\. GuoOpensearch\-sql: enhancing text\-to\-sql with dynamic few\-shot and consistency alignment\.Proceedings of the ACM on Management of Data3\(3\),pp\. 1–24\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p2.1)\.
- Xuet al\.\(2022\)F\. F\. Xu, B\. Vasilescu, and G\. NeubigIn\-ide code generation from natural language: promise and challenges\.ACM Transactions on Software Engineering and Methodology \(TOSEM\)31\(2\),pp\. 1–47\.Cited by:[§1](https://arxiv.org/html/2610.02679#S1.p2.1)\.
- Yuet al\.\(2019\)T\. Yu, R\. Zhang, H\. Er, S\. Li, E\. Xue, B\. Pang, X\. V\. Lin, Y\. C\. Tan, T\. Shi, Z\. Li,et al\.CoSQL: a conversational text\-to\-sql challenge towards cross\-domain natural language interfaces to databases\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 1962–1979\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p2.1)\.
- Zhanget al\.\(2023\)W\. Zhang, Y\. Shen, W\. Lu, and Y\. ZhuangData\-copilot: bridging billions of data and humans with autonomous workflow\.arXiv preprint arXiv:2306\.07209\.Cited by:[§2](https://arxiv.org/html/2610.02679#S2.p1.1)\.